Today in computing mysteries

[sebenikela]

so for some reason my linux laptop hates R. This is confusing because:

- the specs on this machine are similar to my 2015 MacBook Pro, which runs R without issues beyond occasional slowness (same 8GB of RAM, similar processing speed iirc, both SSD drives)

- this machine runs similarly computationally intensive python code quite cheerfully

HOWEVER

- when using RStudio, my R session kept crashing anytime it dealt with large arrays

- so I switched to R in a Jupyter notebook (well, technically Hydrogen in Atom but that’s essentially the same thing) and that froze the whole machine until it was done working

- I have not done anything super complicated in straight up command line R from a terminal, but at this point I doubt that’d make much difference. also, that would be a shitty long-term solution regardless

I have 2 questions for anyone who knows wtf I'm talking about:

1. WHY?

2. Can I make R work better? How?

Alternatively, I could apparently set up a remote notebook server on my mac and access it on my linux laptop, but that sounds like an even bigger pain in the ass.

(relatedly, I configured my mac (which is no longer a laptop due to screen issues that means it needs an external display) to allow me to SSH in remotely, and to test that it was working I installed a terminal emulator (Terminus) on my phone which now means that I can execute R scripts to process multiple gigabytes of data... FROM MY PHONE. Which is the most hackerish thing ever and I am vastly amused by it)

Comments

[elusiveat]

I would start by checking what's going on with resource use on your computer (RAM and CPU) during your R sessions. That can help with troubleshooting performance issues.

R normally loads all data structures into memory, which can take up a lot of memory depending on what kind of analysis you're doing, and once RAM gets filled, the CPU thrashes a lot to write stuff in and out of swap.

I should also note that I'm in the habit of using plain old R (not Rstudio) on my computer, which runs Ubuntu. This *might* be because I ran into some Rstudio glitchiness when I was first starting out and I just found the process smoother using R. One way or another, I seem to be the only data sciencey person I know who uses R, not Rstudio. You might see if that makes a difference.

I believe that when running R without extensions you only get to use one CPU at a time, so with my quad processor I only end up using 25% of memory capacity at a time. That's something else to look at. I believe there are extensions that can divide data up appropriately to be handled on different CPUs (`snow` is the main package for this, I believe: https://www.r-bloggers.com/quick-guide-to-parallel-r-with-snow/) but I haven't figured them out.

Good luck...

[mildred_of_midgard]

I seem to be the only data sciencey person I know who uses R, not Rstudio. You might see if that makes a difference.

I'm only very loosely a data sciencey person, but I also run R on Ubuntu with no problems. The one time I ran Rstudio (because I was trying to come up with instructions for someone else), I didn't have any problems. But I admittedly wasn't doing anything with large data sets on that occasion.

Seconding what elusiveat says, and adding that you can start with the command `top` to check resource usage. Let me know if you're not familiar with it.

One thing that would help: let us know what flavor and release of Linux you're using. Version of R might also help. I personally am on Ubuntu 16.04 and R 3.6.2, and I also have 8 GB of RAM.

Something that may or may not be relevant to you: check out how much memory the nautilus process is using. Nautilus has had what I think is a memory leak bug in all of my Ubuntu releases for the last several years, and if I let it run for too long, it starts consuming way more memory than it needs. On top of something with a big memory footprint like R, that might be enough to set off thrashing.

Good luck!