Procesing big data with Vaex, dataframes and no clusters

Descripción completa

Nowadays data is getting bigger and bigger, making it almost impossible to processed it in desktop machines. To solve this problems, a lot of new technologies(Hadoop, Spark, Presto, Dask, etc.) have emerged during the last years to process all the data using multiple clusters of computers. The challenge is that you will need to build your solutions on top of this technologies, requiring designing data processing pipelines and in some cases combining multiple technologies. However, in some cases we don't have enough time or resources to learn to use and setup a full infrastructure to run a couple experiments. Maybe you are a researcher with very limited resources or an startup with a tight schedule to launch a product to market. A way to archive this is using Vaex, Python library, with a similar syntax to Pandas, the help us to work with large data-sets in machines with limited resources were the only limitation is the size of your hard drive. Vaex provides memory-mapping, so it will never touch or copy the data to memory unless is explicitly requested.