Tables On Disk

Question: Tables can be written to disk as either data files, splayed tables, or partitioned tables. Data files are efficient for small sized tables (<1 million rows), splayed tables are efficient for medium sized tables (<100 million rows), partitioned tables are efficient for large sized tables (>100 million rows). Given the below table, save the table 't' to a file 'data', a splayed table 'splay', and date partitioned table 't' with the date being the date in the data. All of these files/directories should be contained within the same directory.

More Information:

https://code.kx.com/q4m3/14_Introduction_to_Kdb+/#140-overview

Example

                                
                                q)t:([]time:2020.01.01+asc 10?.z.T;sym:10?`AAPL`IBM;px:10?200f;size:10?200j)

/ your statements

q)\l .
q)\ls -l
"total 16"
"drwxr-xr-x 3 alvi alvi 4096 Jun  5 06:38 2020.01.01"
"-rw-r--r-- 1 alvi alvi  344 Jun  5 06:58 data"
"drwxr-xr-x 2 alvi alvi 4096 Jun  5 06:59 splay"
"-rw-r--r-- 1 alvi alvi   17 Jun  5 06:21 sym"
q)select from data / file
time                          sym  px       size
------------------------------------------------
2020.01.01D01:37:00.256000000 IBM  155.4261 0
2020.01.01D02:19:02.023000000 AAPL 16.45375 197
2020.01.01D02:41:42.304000000 AAPL 102.6404 113
2020.01.01D02:41:53.390000000 IBM  98.95657 192
2020.01.01D03:45:55.156000000 AAPL 173.3113 74
2020.01.01D05:24:32.733000000 IBM  128.2995 55
2020.01.01D06:09:51.007000000 AAPL 181.6542 150
2020.01.01D06:36:18.791000000 IBM  195.9219 51
2020.01.01D06:56:52.410000000 AAPL 61.54981 4
2020.01.01D06:58:25.413000000 AAPL 73.04546 46
q)select from splay / splayed
time                          sym  px       size
------------------------------------------------
2020.01.01D01:37:00.256000000 IBM  155.4261 0
2020.01.01D02:19:02.023000000 AAPL 16.45375 197
2020.01.01D02:41:42.304000000 AAPL 102.6404 113
2020.01.01D02:41:53.390000000 IBM  98.95657 192
2020.01.01D03:45:55.156000000 AAPL 173.3113 74
2020.01.01D05:24:32.733000000 IBM  128.2995 55
2020.01.01D06:09:51.007000000 AAPL 181.6542 150
2020.01.01D06:36:18.791000000 IBM  195.9219 51
2020.01.01D06:56:52.410000000 AAPL 61.54981 4
2020.01.01D06:58:25.413000000 AAPL 73.04546 46
q)select from t where date=2020.01.01 / partitioned
date       time                          sym  px       size
-----------------------------------------------------------
2020.01.01 2020.01.01D01:37:00.256000000 IBM  155.4261 0
2020.01.01 2020.01.01D02:19:02.023000000 AAPL 16.45375 197
2020.01.01 2020.01.01D02:41:42.304000000 AAPL 102.6404 113
2020.01.01 2020.01.01D02:41:53.390000000 IBM  98.95657 192
2020.01.01 2020.01.01D03:45:55.156000000 AAPL 173.3113 74
2020.01.01 2020.01.01D05:24:32.733000000 IBM  128.2995 55
2020.01.01 2020.01.01D06:09:51.007000000 AAPL 181.6542 150
2020.01.01 2020.01.01D06:36:18.791000000 IBM  195.9219 51
2020.01.01 2020.01.01D06:56:52.410000000 AAPL 61.54981 4
2020.01.01 2020.01.01D06:58:25.413000000 AAPL 73.04546 46
                                
                            

Solution

Tags:
tables
Searchable Tags
algorithms api architecture csv data structures dictionaries disk feedhandler finance functions ingestion ipc iterators machine learning math multithreading optimizations realtime sql statistics streaming strings tables temporal websockets

Email sent!

Email not sent