Hi all,
I'm Richard from the Insight team, and I've been leading our work on the analytics project. I just wanted to post a quick update about the ETL tool we've been working on ahead of the developer call this week.
We'll be giving a brief demo of the tool, both as a CLI exporter and how you can manipulate the data once it's been loaded into a SQL DB. You can see the slides for the demo below.
https://docs.google.com/presentation/d/1wmgfddbD-shKoP0b32wQeaXramVhfo0Ayq1nvc65Obg/edit?usp=sharing
If you have any questions either before our after our demo, either post a comment on the forum post (https://forum.icon.community/t/icon-analytics-update/1133), or send us a message on Telegram (Richard: @shinyfoil; Rob: @robcio)
For information, I’d suggest taking a look at Aegis 1, which does something similar (but not maintained anymore), but more importantly, Chainalytic 2, which is much faster than reading blocks through a HTTP provider as it reads directly from leveldb with plyvel. @duyyudus
Reply:
Thanks for these references. I wasn't personally aware of Aegis, but I do know of Chainalytic. We'd previously evaluated Chainalytic as a potential place to increase our speed, however, when using this package in combination with Airflow, the LevelDB parsing approach doesn't work. One of our requirements in this iteration of the package was that it be able to be a drop-in replacement for any of the other blockchain-etl packages to support that architecture. However, bringing in the ability to parse the LevelDB directly is on our radar and would be a feature we would consider adding in a future version.