Jackson Kwok, Infrastructure Steering Committee (ISC) recipient for the 2020 cycle, discusses the project Sketch and its implementation in 2021 as an R Package in the CRAN environment. The Sketch package was developed with the goal of having a package that translates R to Javascript. With this experience, Jackson then later developed the package, Animate, which requires no prior knowledge of JavaScript. Jackson discusses Sketch’s origins in JavaScript (JS) and data visualization, its potential applications, and the level of expertise required to utilize the package effectively.
RC: Where did you come up with the idea of a program that translates R into JS?
JK: It started with a GitHub issue by Jonathon Carroll at the rOpenSci R OzUnconference 2017. The idea was to use the JS library P5 for visualization in R, and a few of us came together to work on a prototype package called “realtime”. After the hackathon, I continued to pursue the idea. As I was reading the book The Nature of Code, I realized it took only a few rewriting rules to translate JS to syntactically correct R code. After some manual experiments, I wrote the package to test the idea (in the reverse direction transpiling R to JS) and it worked! The package got its name because P5 refers to the digital drawings as Sketches; I named the folder ‘Sketch’ when I studied P5 and later used the same folder for the package. There is nothing specific to P5 that makes the conversion work. I tried other JS libraries and it also worked very well, so I refactored the package into a general purpose R-to-JS transpiler.
Examples are of Physics engine and 3D models on Sketch
RC: How can I use Sketch in a shiny app?
JK: On the documentation website, there is a page on how to use Sketch on a Shiny app. Once you develop your R script and transpile it into JS, you can include that in the Shiny app as usual.
RC: How much JS knowledge do you need to use this package?
JK: A little bit to get started. You don’t need to know the syntax, but you need to know how JS works because it works differently from R. Vector and List in R correspond roughly to Array and Object in JS. Both of them are passed by reference rather than passed by value, and Array uses 0-based indexing. That is what is needed to get started. The Pitfalls section of the documentation has the complete list.
There are some tutorials in the Tutorial section that users can follow with no knowledge of JS. They will be able to pick up some JS along the way. What Sketch does is to let users call JS libraries like R packages. You still need to learn the commands in the package to use it, just like any new R package.
RC: Can I use different R Packages with Sketch? (aka, could I use rQTL to analyze and visualize the data?)
JK: Yes. That was one of the key milestones in the 2nd proposal of this project. Sketch uses WebSocket to establish a connection between R and the browser so that you can use any R package to perform some calculations and then pass the results to the browser. The connection is live, so you can also perform operations in the browser and get the corresponding data back to R. It is very handy if you need some interaction that is very hard to express in code. For instance, if you need to select something using a lasso tool (irregular shape) it will be much easier. You just draw it rather than have to figure out the coordinates.
If you choose to use an R package with Sketch, then the application will no longer be stand-alone. A targeted use case is to let users add a customizable domain-specific interface to an existing analysis performed using many R packages. If you don’t use any R package at all, then the Sketch application can be deployed as a stand-alone website.
RC: What is the progress on Sketch?
JK: The second proposal is now complete with many new features added. Among other things, there is the support of R6-style of OOP which helps you structure larger programs, the WebSocket which lets you do bidirectional updates, and a knitr engine for RMD and publish support. On the package side, I do not see substantial changes from here on; the later updates will mostly be fixes and patches.
One interesting direction that I have been looking into is to transpile R to AssemblyScript, which in turn can be compiled into WebAssembly. I have done some preliminary studies, and it seems this path is viable. AssmeblyScript is still maturing, so this will not be worked into Sketch for now, but I will keep an eye on it.
The next thing Sketch looks to expand on is use cases. Lately, I found out it is not difficult to use Sketch to create a web-based graphics device for animated plots in R, so I have been working on it, and the results are encouraging. Another thing I discovered is that it is quite easy to go from ggplot2 to rayshader to VR, and it works well with 3d histograms on maps. Also, another application that came by surprise is that with Sketch, you can control Excel from R. Excel has a JavaScript API and supports WebSocket protocol. As Sketch transpiles R to JS and speaks WebSocket, it turns out controlling Excel from R just works out fine!
I discover new possibilities with Sketch every now and then, and that makes me realize this work is really a good step in strengthening the R-Web integration and expanding the R application landscape.
The ISC project was delivered in early 2021. Since its release date, we now have a package called Animate, on CRAN. It would have been difficult to build without the Sketch package since it uses a heavy amount of JavaScript. Animate provides animated diagrams beyond XY plots. Unlike Sketch which needs the user to have some background knowledge of integrating JavaScript, with Animate, users can manipulate graphical elements on the screen using native R commands without knowing the animations are powered by JavaScript.
RC: Has Sketch expanded the scope of R visualizations for the R community?
JK: It has definitely expanded the scope for JavaScript and R. The Sketch website features a showcase page showing some of the new possibilities, including more advanced 3D model animations and agent-based visualizations. In general, Sketch is well-suited for cases in which you need to use a JavaScript library beyond direct API calls or where the API has an imperative style.
Looking back after two years of completing the package, I think Sketch succeeded in producing executable transpiled JavaScript and exploring how far one can control JavaScript with R, but it fell short in abstracting away the JavaScript side of things. These shortcomings are addressed by Animate.
RC: How did you get involved?
JK: I first got into interactive visualizations back in 2016 after seeing a few great talks and demos online, e.g. the Invention on Principle, Stop Drawing Dead Fish by Bret Victor, the Parable of the polygons by Vi Hart and Nicky Case and the ConvNetJS by Andrej Karpathy. They got me started learning JS, but I am an R user at heart. I wanted to make interactive visualization in R, then the R OzUnconference came, and you know the rest. Looking back, it has been quite a journey picking up the skills needed to deliver this project.
I found out about the R consortium ISC program when the research fund that I was under ran out early. I reckoned it was a great opportunity to contribute to the R community and get some financial support, so I put in an application with Kate Saunders, a good friend who loves data visualization and R programming with expertise in spatial statistics – one of the key areas I want Sketch to develop into.
RC: What was your experience working with the R Consortium? Would you recommend applying for a grant to others?
JK: I had a great experience and highly recommend it to others looking to develop packages that can help the R community. I personally learned a lot about the grant application process, like writing a proposal, arguing for the benefits of the project, and doing deeper research on what you want to solve and how your solution can be successful.
The process was also great for picking up medium- to long-term software planning, like structuring the development with milestones, which I previously had no experience with. Overall it has been a great learning and rewarding experience! What helped me plan the proposal were the guidelines provided by the R Consortium. I was able to take bigger infrastructure problems and group them into small solvable groups of tasks.
RC: What do you do for your day job?
JK: I am finishing up my postdoc at the St. Vincent’s Institute of Medical Research. My research group is working on a translational project called BRAIx. It’s about transforming breast cancer screening in Australia using AI. In the project, I have been using Sketch to create customized data visualization tools for data and model diagnostics.
About ISC Funded Projects
A major goal of the R Consortium is to strengthen and improve the infrastructure supporting the R Ecosystem. We seek to accomplish this by funding projects that will improve both technical infrastructure and social infrastructure.