[2022] 30 Day Map Challenge -- FAILED

🧰 Publicly available data

Hunting for spatial data comes naturally now. There seems to be less and less opportunity for doubts when we could attach a pair of coordinates to some places.

For work and hobby, hunting for data take almost half of the usable hours I set aside to execute certain objectives; if not 100%. Although the internet is a vast plain of data, not all of them are usable. The democratization of data is a subject that is to translucent to discuss but to solid to argue with. Thus, with differing opinions, we get different versions of them online. Here are some of the interesting data platforms I manage to scour based on their thematic subject

🌳 Nature and Environment

Delta at Risk - Profiling Risk and Sustainability of Coastal Deltas of the World. I found this while lamenting on how people love asking for data addition into their maps at the eleventh hour. I find their confidence in my skills quite misleading but flattering nonetheless. But it does not make it any less troublesome.

Protected Planet - Discover the world's protected and conserved areas. This platform includes not just data of protected areas, but also other effective area-based conservation measures like ICCAs IUCN listing and as the website claims, it is updated regular via submissions from agencies. So far, I found this platform to be the most convenient since it rounds up all possible conservation-based themes which also includes World Heritage Sites.

Global Forest Change (2000-2020) - The global forest extent change since 2000 to the current year or lovingly referred to as the Hansen data by most forestry RS specialist. This data is updated annually and to be honest, the platforms are literally everywhere. But this platform is legitimate under Earth Engine Apps and you can refer to Google Earth Engine for future data updates to ease your search.

👩‍⚖️ Administrative Data

GADM - Map and spatial data for all countries and their sub-divisions.

🏦 Built-environment Data

OpenStreet Map - This database is the most amazing feat of tech-aware crowdsourcing. A little more than 2 decades ago, some 'experienced' gate-keeping professionals would have refuted its legitimacy within an inch of their lives but OSM has proven that time prevails when it comes to bringing the accessibility and network data into practical use. I am not that adept with downloading from this website so I go directly to a more manual data download. My favorite is the Geofabrik Download but you can also try Planet OSM.

🎮 Other Cool Data

OpenCell ID - Open database platform of global cell towers. Cleaning the data is a nightmare but I think it is just me. I have little patience for cerebral stuff.

So, those are some of the data I managed to dig for personal projects. Hope it helps you guys too!

In ArcGIS Pro, the Erase tool only comes with the Advanced license. There are other ways to go about removing parts of a polygon/line data layer like the Clip tool. But Union is that tool where it makes more sense by principle.

It works by marking overlapping parts of two different data layer with integers; 1, 2 and so forth. Those that do not overlap is universally -1. So, remove everything else that you want out of the picture by deleting output features that contain FID integer values of more than -1! Simple eh?

Check out the <3 minutes demo below!

P/S: Happy New Year peeps! ♥

diy geo app

Esri has been releasing more and more MOOC over the span of 2 years to accommodate its increasingly large expanse of products within the ArcGIS ecosystem. 

But of all the MOOCs that I've participated in, 'Do-It-Yourself Geo App MOOC' must be the most underrated ones produced by Esri Training. The functionalities highlighted within the MOOC took the anthem right off their recent Esri UC 2020 that went virtual. The curriculum includes:

The creation of hosted feature layer (without utilizing any GIS software medium like ArcMap or ArcGIS Pro).

The basics of the ArcGIS Online platform ecosystem:

hosted feature layer >  web map > web app 

Basically, to view a hosted feature layer, you will need to drag it onto a 'Map' and save it as a web map.

Conventionally, web map suffices for the visualization and analytical work for the likes of any geospatialist who are familiar with Web GIS. 

But this time, Esri is highlighting a brand new web map product called 'Map Viewer Beta'. Why beta? Cause it is still in beta version but so sleeky cool that they just had to let every have a shot at using it. Truth be told, Map Viewer Beta did not disappoint.

Even so, Map Viewer Beta still has some functionalities that have yet to be implemented. 

Using web map to visualize data, configure pop-up, execute simple analysis and extending it to Map Viewer Beta interface 

Utilizing Survey123 for crowdsourcing data; the first level of citizen science and creating a webmap out of it.

Creating native apps using AppStudio for ArcGIS; no coding required. 

Some tidbits on accessing the ArcGIS API for JavaScript

I love how cool it is that this MOOC actually shows you step-by-step on how to use the new Map Viewer Beta and explain the hierarchy of formats for the published content in the ArcGIS Online platform

I have established my understanding of ArcGIS Online ecosystem 3 years back but I do find it awkward that such powerful information is not actually summarized in a way that is comprehensible for users that have every intention of delving into Web GIS. And Web GIS is the future with all the parallel servers that could handle the processing/analysis of large amount of data. ArcGIS Online is a simplified platform that provides interfaces for the fresh-eyed new geospatial professionals. 

It is quite well-know for the fact that there has been some criticism as to the domination of Esri within the GIS tools/resources within the geospatial science industry, but I believe it is something we could take as a pinch of salt. Not everything in Esri's massive line of commercial products are superior to other platforms but it is a starting point for any new geospatialists who wants to explore technologies there are not familiar with. 

All in all, this MOOC is heaven-sent. For me, I have been playing with the web apps and web maps for close to 4 years and I can attest to the fact that it covers all the basics. For the developer's bit, maybe not so much as going through it in a distinct step-by-step but it does stoke the curiosity as to how it works. The question is, how do we make it work. Now that's a mystery I am eager to solve. 

I'm going to put this on my ever-expanding to-do list and think JavaScript for another few more months of testing out this ArcGIS API for JavaScript implementation. Tell me if you wanna know how this actually works and I'll share what I find out when I do.

For those who had missed out on this cohort, fear not. This MOOC runs twice a year and the next cohort is going to be from Feb 17 to March 17 2021. The registration is already open, so don’t hold back and click the link below:

Do-It-Yourself Geo Apps

Do register for a public account before signing up or just click 'Register' at the MOOC's page and it's open the open to either sign in or 'Create a public account'. It was a blast and I'm sure, if you've never used any of the feature I've mentioned above, you'll be as wide-eyed as I was 3 years ago. :D

Till then, stay spatially mappy comrades!

P/S: If you complete all the assignments and quizzes, you'll get a certificate of completion from Esri. Which is pretty rad!

With this, I am commencing my submission for the #30DayMapChallenge for 2023 🗺

The categories outlined is similar to that of last year but I am never going to hate this repetition. How can I? It's a basics of making maps and there's so much to learn from the single-word theme.

Any aspiring map-makers out there? Let's share our maps for this wonderful month of November under the #30DayMapChallenge 2023!

Python: Geospatial Environment Setup (Part 2)

Python: Geospatial Environment Setup (Part 2)

Hey again folks! I am here for the second part of Python environmental setup for a geospatial workspace. I published the first part of this post two weeks ago. So if you've not yet read that, I'll catch you up to speed with our checklist:

Install Python ☑

Install Miniconda ☑

Install the basic Python libraries ☑

Create a new environment for your workspace

Install geospatial Python libraries

🗃 Create a new environment for your workspace

Since we have actually manually set up our base environment quite thoroughly with all the basic libraries needed, to make our work easier, we can just clone the base environment and install all the additional essential libraries needed for geospatial analysis. This new environment will be called geopy. Feel free to use a name you identify most with.

Why don't we just create a new environment? Well, it means we have to start installing the Python libraries again from scratch. Although it is no trouble to do so, we want to avoid installing so many libraries all at once. As I mentioned in Part 1, there is always a risk where incomplete dependencies in one library will affect the installation of other libraries that you intend to install in one go. Since we already have a stable and usable base environment, we can proceed to use it as a sort of pre-made skeleton that we will build our geospatial workspace with.

1️⃣ At the Anaconda Command Prompt, type the following:

2️⃣ Press Enter and the environment will be clone for you. Once it is done, you can use the following command to check the availability of your environment 👇🏻

You should be able to see your geopy environment listed along with the base environment.

👩🏻‍💻 Install geospatial Python libraries

Here we will proceed with the installation of a few geospatial Python libraries that are essential to reading and exploring the vectors and rasters.

🔺 fiona: This library is the core that some of the more updated libraries depend on. It is a simple and straightforward library that reads and writes spatial data in the common Python IOs without relying on the infamous GDAL's OGR classes.

🔺 shapely: shapely library features the capability to manipulate and edit spatial vector data in the planar geometric plane. It is one of the core libraries that recent geospatial Python libraries rely on to enable the reading and editing of vector data.

🔺 pyproj: is the Python interface for the cartographic projections and coordinate system libraries. Another main library that enables the 'location' characteristics in your spatial data to be read.

🔺 rasterio: reads and writes raster formats and provides a Python API based on Numpy N-dimensional arrays and GeoJSON.

🔺 geopandas: extends the pandas library to allow spatial operations on the geometric spatial data i.e shapefiles.

💀 As you might have noticed, we won't be doing any direct gdal library installation. It's mainly due to the fact that its installation is a process that seems to be accompanied by misery at every turn and involved workarounds that are pretty inconsistent for different individuals. Does it mean that we won't be using it for our Pythonic geospatial analysis? Heck no. But we will be taking advantage of the automatic dependency installation that comes with all the libraries above. The rasterio library depends on gdal and by installing it, we integrate the gdal library indirectly into our geospatial environment. I found that this method is the most fool-proof. Let's proceed to the installation of these libraries.

1️⃣ At the Anaconda Command Prompt, should you start from the beginning, ensure that your geopy environment is activated. If not, proceed to use the following command to activate geopy.

Once activated, we can install the libraries mentioned one after another. Nevertheless, you also have the option of installing them in one go directly using a single command 👇🏻

💀 geopandas is not included in this line-up NOT because we do not need it. It's another temperamental library that I prefer to isolate and install individually. If gdal is a rabid dog...then geopandas is a feral cat. You never know how-when-why it doesn't like you and forces a single 10-minute installation drag to hours.

3️⃣ Once you're done with installing the first line-up above, proceed with our feral cat below 👇🏻

4️⃣ Use the conda list command again to check if all the libraries have been installed successfully.

🎉Et voilá! Tahniah! You did it!🎉

🎯 The Jupyter Notebook

It should be the end of the road for the helluva task of creating the geospatial environment. But you're going to ask how to start using it anyway. To access this libraries and start analyzing, we can easily use the simple and straight-forward Jupyter Notebook. There are so many IDE choices out there but for data analysis, Jupyter Notebook suffices for me so far and if you are not familiar with Markdown, this tool will ease you into it slowly.

Jupyter Notebook can be installed in your geopy environment as follows:

And proceed to use it by prompting it open via the command prompt

It ain't that bad, right? If you're still having problems with the steps, do check out the real-time video I created to demonstrate the installation. And feel free to share with us what sort of problems you have encountered and the workaround or solutions you implemented! It's almost never a straight line with this, trust me. As mentioned in the previous post, check out the quick demo below 👇🏻

See you guys again for another session on geospatial Python soon!

take care

Studying is hard enough when you are fighting the onset of hormonal hurricane of emotions while you're a teenager. Then you have you formative 5 to 6 years of collegiate years where you learn that your idealistic notions of the world is just plain snobbish and that you're one of the many that finds satisfaction in pushing your ideas into other people's head. Doing all that while studying, it was hard too.

Then comes the job hunting, climbing ladders to nowhere from one workplace to another...just drifting. That was the vacation off from studying and just wanting to find stability.

Then one day, you broke your heart at the same time you get an opportunity to a decent work somewhere. You ran. Then it was about understanding this whole new dimension of knowledge that you've fallen in love with. It was like your first love came back to find you and tell you he's back and that he won't leave you alone anymore. So, back to the books you fall into.

So yeah...studying is a rollercoaster of emotions. Learning is a lifelong journey. You're never too old to learn something new. And then phrase where you can't teach old dogs new tricks is just plain inaccurate. You can teach them new tricks and they can do it if they want to. It's the matter of either they want to do it or if their body can still keep up with them.

If you're an older student, I feel you. You feel left behind at times and frustrated because you feel like you can't catch up with things. If you're someone who's starting to work on research studies, it may be a harder state of mind. Because you're almost always on your own lamenting on you're comprehension of theories and questioning if you get whatever you absorbed right. So...it is hard. How do you know that you're alright and that you're not spinning down the road of doomed isolation? You don't. These are some of the glaring things I think are the universal problems among people who, with the courage of a lion, answered the call of hermitage and subject themselves to an occupation so feared that it is deemed over once high school ends; student:

Wardrobe reduction to daily uniform of hoodies/sweaters and jog-pants/sweatpants

It's totally ok. If you're wearing the same one everyday for the rest of the week, you still win. It's practical, it's easy and for hijabsters, the hoodie helps with instant full head cover while you run to receive those stuff you endlessly ordered online. There's nothing to be ashamed off.

💡I say: Shower 🚿. Every. Day. Twice a day is most ideal. Stay fresh and hygienic. If you're not going anywhere nor are you working out in those attire, then you're good. At least for a week. When you shower, you're giving your body a break from the exhausting task of slouching, the heat of your brain going overdrive and your own face breaking out from the stress. So yeah...shower. Wash your face. Brush your teeth. You're gonna be at risk of letting yourself go when you're left alone too comfortable in your own company. If you're an introvert, it feels like heaven but it doesn't mean you should be at the risk of losing out on the joys of prepping yourself to look good once in a while. Regularly make the effort to find decent clothes you'll use to go out on a Saturday window shopping and kick back to relax. On the rare occasion you get a break from your endless paper chasing, Zoom calls and writing, you can use your wardrobe clearing day to match few clothing and have a runway at home yourself or for a date with yourself to the Netflix movie you're gonna watch on your laptop. Who says you can't dress up to do online shopping? That's the only way you'll know that you do not need that new shade of Forencos lipstick you saw the other day. Duh.

Sudden weight gain/loss

I can't say that is unexpected especially if you're the kind who can go long without exercises and the type who munch while you work (which is a bad habit by the way). But sometimes you can't help if especially if you live on your own or have your own living space. Being on your own makes it kind of easy to get off-track about the norm of things; how much have you eaten today...is that your 5th coffee...should you be finishing that leftovers...etc. It is harder if you don't have your own kitchen and rely on packed food or processed ones.

💡I say: I can assure you, it is NOT unusual. This happens often. Each one of us have different eating habits. There isn't a flexible advise to cater this problem. But it helps to think that when you're setting up a study environment, you're setting up a living environment that enable you to be inspired enough for productivity. I make it a point to have food available but at a good distance and exercise attire/equipment nearby. Better yet, wear them. To curb that crazy cravings, I try my best to have my meal on-time; breakfast at 8 am, lunch at 12 pm or 1 pm etc. Eat what you love, but in moderation. Hydrate regularly. It's hard, I know. All the chips🍟 and chocolate chip cookies🍪 and sweetened 3-in-1 lattes☕...I just...well, just remember...in moderation and you're a human. Don't beat yourself up if you eat them anyway.

Back pain, knee pain, frozen shoulder...everything hurts

I have a hard time tearing away from what I am doing for the 20-minutes-work-5-minutes-rest rule but I try my best to maintain good posture. The bad habit of crouching and slouching at desk jobs is a universal problem. At the onset of the pandemic, for the first time in my life, my back felt strained. I panicked and started to seriously stretch myself and found that not only I am killing my back, I was not breathing properly when I am doing my work due to my posture.

💡I say: Don't break the bank trying to buy new furniture or tools because from experience, you end up not actually using them for the purpose you bought them in the first place. Eg. treadmill...you're more likely to hang your clothes or towels there...or super expensive yoga mats. Carpet works just fine too.

Standing desk is deemed to have little to no effects on spinal/muscle health by some studies but I went on ahead and made a make-shift standing desk. I found that I'm more alert when I work standing and breathe better. There was no contest...standing desk is a choice made. My standing table is an old dining table that I turned into my bed where I put my portable laptop desk. So, it also works as a barre that I used to stretch my legs for that mid-splits I've been working on.

These are some of the stuff that plague me daily and how I cope. How about you guys? How do you take care of yourself as someone who willingly serve yourself on a platter to tertiary educational torture to console your ever-thirsty curious mind?

zero to pandas

There are alot of Python courses out there that we can jump into and get started with. But to a certain extent in that attempt to learn the language, the process becomes unbearably long and frustratingly slow. We all know the feeling of wanting to run before we could learn how to walk; we really wanna get started with some subtantial project but we do not know enough to even call the data into the terminal for viewing.

Back in August, freeCodeCamp in collaboration with Jovian.ai, organized a very interesting 6-week MOOC called Data Analysis with Python: Zero to Pandas and as a self-proclaimed Python groupie, I pledged my allegiance!

If there are any expectation that I've managed to whizz myself through the course and obtained a certificate, nothing of that sort happened; I missed the deadline cause I was busy testing out every single code I found and work had my brain on overdrive. I can't...I just...can't. Even with the extension, I was short of 2 Pythonic answers required to earn the certificate. But don't mistake my blunders for the quality of the content this course has to offer; is worth every gratitude of its graduates!

Zero to Pandas MOOC is a course that spans over 6 weeks with one lecture webinar per week that compacts the basics of Python modules that are relevant in executing data analysis. Like the play on its name, this course assumes no prior knowledge in Python language and aims to teach prospective students the basics on Python language structure AND the steps in analyzing real data. The course does not pretend that data analytics is easy and cut-corners to simplify anything. It is a very 'honest' demonstration that effectively gives overly ambitious future data analysts a flick on the forehead about data analysis. Who are we kidding? Data analysis using programming language requires sturdy knowledge in some nifty codes clean, splice and feature engineer the raw data and real critical thinking on figuring out 'Pythonic' ways to answer analytical questions. What does it even mean by Pythonic ways? Please refer to this article by Robert Clark, How to be Pythonic and Why You Should Care. We can discuss it somewhere down the line, when I am more experienced to understand it better. But for now, Packt Hub has the more comprehensive simple answer; it simply is an adjective coined to describe a way/code/structure of a code that utilizes or take advantage of the Python idioms well and displays the natural fluency in the language.

The bottom line is, we want to be able to fully utilize Python in its context and using its idioms to analyze data.

The course is conducted at Jovian.ai platform by its founder; Aakash and it takes advantage of Jupyter-like notebook format; Binder, in addition to making the synchronization available at Kaggle and Google's Colab. Each webinar in this course spans over close to 2 hours and each week, there are assignments on the lecture given. The assignments are due in a week but given the very disproportionate ratio of students and instructors, there were some extensions on the submission dates that I truly was grateful for. Forum for students is available at Jovian to engage students into discussing their ideas and question and the teaching body also conducts office hours where students can actively ask questions.

The instructor's method of teaching is something I believe to be effective for technical learners. In each lectures, he will be teaching the codes and module requires to execute certain tasks in the thorough procedure of the data analysis task itself. From importing the .csv formatted data into Python to establishing navigation to the data repository...from explaining what the hell loops are to touching base with creating functions. All in the controlled context of two most important module for the real objective of this course; Numpy and Pandas.

My gain from this course is immensely vast and that's why I truly think that freeCodeCamp and Jovian.ai really put the word 'tea' to 'teachers'. Taking advantage of the fact that people are involuntarily quarantined in their house, this course is something that should not be placed aside in the 'LATER' basket. I managed to clear my head to understand what 'loop' is! So I do think it can solve the world's problem!

In conclusion, this is the best course I have ever completed (90%!) on data analysis using Python. I look forward to attending it again and really finish up that last coursework.

Oh. Did I not mention why I got stuck? It was the last coursework. We are required to demonstrate all the steps of data analysis on data of our choice, create 5 questions and answer them using what we've learned throughout the course. Easy eh? Well, I've always had the tendency of digging my own grave everytime I get awesome cool assignments. But I'm not saying I did not do it :). Have a look-see at this notebook and consider the possibilities you can grasp after you've completed the course. And that's just my work...I'm a standard C-grade student.

And the exciting latest news from Jovian.ai is that they have upcoming course at Jovian for Deep Learning called Deep Learning with PyTorch: Zero to GANS! That's actually yesterday's news since they organized it earlier this year...so yeah...this is an impending second cohort! Tentatively, the course will start on Nov 14th. Click the link below to sign-up and get ready to attack the nitty-gritty. Don't say I didn't warn ya.

And that's me, reporting live from the confinement of COVID pandemic somewhere in a developing country at Southeast Asia....

train on water