Some large datasets are pushing memory and some functions I’m writing to the limit. I wanted to ask some questions about subsetting, of matrices and arrays in particular:...
+1 for parquet and arrow. If you’re pushing memory better to just treat it as a completely out of memory problem. If you can split the data into multiple parquet files with hive style or directory partitioning it will be more efficient. You don’t want parquet files too small though (I’ve heard people saying 1 GB each file is ideal, colleagues at work like 512 MB per file - but that’s on an AWS setup).
Bonus is once you’ve learned the packages it’ll be the same for all out of memory big datasets.
Re: c) I will be a dirty shill for VSCode and R lol, example here. I find it much better for R shiny development, projects with multiple people and projects with multiple languages. Notebook support is less good out of the box, you will have to get a jupyter kernel set up - but I use scripts more so than notebooks anyway.
Anyway, onto the question! Base R. Yeah, I said it! Whenever I have a weird enough situation where tidyverse functions won’t work due to poor quality data, then I shed a single solemn tear and quietly wish I had done the project in python as I start writing a for loop in what will no doubt be the most hacky solution ever.
Just getting the hang of it. Jerboa app for android is very nice to have - sometimes find it hard to search up a community in another lemmy server even when I know the same (I think the search is case sensitive?). That's on the mobile web interface... Haven't figured out how to find a new community on jerboa itself yet
Does subsetting (matrices or arrays) always perform a partial copy?
Some large datasets are pushing memory and some functions I’m writing to the limit. I wanted to ask some questions about subsetting, of matrices and arrays in particular:...
print to console tables that can be easily copied and pasted to Excel
There is a function format_csv in package readr, which outputs csv formated output to string. It can be used as...
What are the things you dislike about R?
I will start:...
Welcome reddit refugees!
how are yall feeling about the website?