index.xml

<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Adrián Abreu</title><link>https://adrianabreu.com/</link><description>Recent content on Adrián Abreu</description><generator>Hugo</generator><language>es-ES</language><copyright>2017-2024 Adrián Abreu powered by Hugo and Kiss Theme</copyright><lastBuildDate>Tue, 27 Jan 2026 11:40:43 +0000</lastBuildDate><atom:link href="https://adrianabreu.com/index.xml" rel="self" type="application/rss+xml"/><item><title>Stuck connecting snowflake integrated mcp and cursor</title><link>https://adrianabreu.com/blog/2026-01-27-stuck-snowflake-mcp-cursor/</link><pubDate>Tue, 27 Jan 2026 11:40:43 +0000</pubDate><guid>https://adrianabreu.com/blog/2026-01-27-stuck-snowflake-mcp-cursor/</guid><description>&lt;p&gt;Lately I&amp;rsquo;m not exactly enjoying my job. Snowflake step curve from a data engineer perspective is really hard.&lt;/p&gt;
&lt;p&gt;Yesterday my boss asked me to test the Snowflake integrated MCP from Cursor. He’d been looking into Snowflake OAuth access, so I went to check the resources. Once again—as of January 27th, 2026—the Terraform resource for this still doesn&amp;rsquo;t exist (though the data source does, for some reason).&lt;/p&gt;
&lt;p&gt;For connecting to snowflake you need to define a &lt;a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-agents-mcp#set-up-oauth-authentication"&gt;security integration&lt;/a&gt;.&lt;/p&gt;</description></item><item><title>Snowflake questionable choices</title><link>https://adrianabreu.com/blog/2026-01-14-snowflake-questionable-choices/</link><pubDate>Wed, 14 Jan 2026 20:14:10 +0000</pubDate><guid>https://adrianabreu.com/blog/2026-01-14-snowflake-questionable-choices/</guid><description>&lt;p&gt;I switched jobs all over again. I&amp;rsquo;m working in a data company called Shalion, where we provide insights about retail products online. If you&amp;rsquo;re brave enough to handle Snowflake&amp;rsquo;s quirks, &lt;a href="https://shalion.teamtailor.com/jobs/6328344-data-engineer"&gt;we&amp;rsquo;re hiring&lt;/a&gt;&amp;hellip;&lt;/p&gt;
&lt;p&gt;I&amp;rsquo;m working on revamping our permission grant system from &lt;a href="https://gitlab.com/gitlab-data/permifrost"&gt;permifrost&lt;/a&gt; to a terraform managed approach.&lt;/p&gt;
&lt;p&gt;We also decided to embrace a 2 layer permission grant, using Access Roles (AR) and Functional Roles (FR). The former has access to the resources with the typical permissions (SELECT, USAGE, CREATE)&amp;hellip;, while the latter is the one that we assign to people and just groups access roles.&lt;/p&gt;</description></item><item><title>Aprender de nuevo</title><link>https://adrianabreu.com/blog/2026-02-13-learn-to-learn-again/</link><pubDate>Tue, 06 Jan 2026 20:14:10 +0000</pubDate><guid>https://adrianabreu.com/blog/2026-02-13-learn-to-learn-again/</guid><description>&lt;p&gt;2025 ha sido un año bastante loco.&lt;/p&gt;
&lt;p&gt;Pasé la mayor parte del año sentando las bases de una startup en el mundo de la seguridad, haciendo frontend, backend, mucho devops y&amp;hellip; poco data.&lt;/p&gt;
&lt;p&gt;Me pilló de lleno la era de la AI y como siempre, el campo ajeno es más verde. Viví la burbuja de v0 y mi trabajo respecto a este, fue bastante miserable. El proyecto a medida que crecía dejaba de ser funcional. Un cambio manual que arreglase el mismo era incompatible.&lt;/p&gt;</description></item><item><title>I’m Building Stuff – My New Motto</title><link>https://adrianabreu.com/blog/2025-01-13-building-stuff/</link><pubDate>Mon, 13 Jan 2025 20:07:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2025-01-13-building-stuff/</guid><description>&lt;p&gt;For the past seven years, I worked in data, and I have mixed feelings about it. I still believe data is the most important part of any app, but it’s meaningless without the app itself.&lt;/p&gt;
&lt;p&gt;Now that I’m working at a startup, I’ve decided to focus on building things. To start, I revisited one of my older projects: a PDF parser about professor designations in the Canary Islands, where one of my best friends works as a teacher.&lt;/p&gt;</description></item><item><title>Finding pet projects</title><link>https://adrianabreu.com/blog/2024-08-13-finding-pet-projects/</link><pubDate>Tue, 13 Aug 2024 07:00:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2024-08-13-finding-pet-projects/</guid><description>&lt;p&gt;As my company undergoes layoffs, I&amp;rsquo;m back on the job hunt. While I&amp;rsquo;m in the field of data, I often find myself missing the hands-on experience that comes from personal projects. I realized that I&amp;rsquo;m not practicing all the skills I need.&lt;/p&gt;
&lt;p&gt;During a recent interview, I was asked about my experience with sending reports via email—something I hadn’t done in a few years. That got me thinking: could I turn this into a pet project?&lt;/p&gt;</description></item><item><title>Developing on windows</title><link>https://adrianabreu.com/blog/2024-03-22-developing-on-windows/</link><pubDate>Fri, 22 Mar 2024 18:06:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2024-03-22-developing-on-windows/</guid><description>&lt;p&gt;Over the years, I&amp;rsquo;ve been using MacOS at work and Ubuntu at home for my development tasks. However, my Lenovo P1 Gen 3 laptop didn&amp;rsquo;t work well with Linux, leading to frequent issues with the camera and graphics (screen flickering, I&amp;rsquo;m looking at you, and it hurts).&lt;/p&gt;
&lt;p&gt;I&amp;rsquo;ve triend Windows Subsystem for Linux (WSL) but it was quite bad to be honest. But as I&amp;rsquo;ve heard of WSL2 and WSLg, I decided to give it another shot.&lt;/p&gt;</description></item><item><title>Querying the databricks api</title><link>https://adrianabreu.com/blog/2024-01-26-querying-the-databricks-api/</link><pubDate>Fri, 26 Jan 2024 09:06:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2024-01-26-querying-the-databricks-api/</guid><description>&lt;p&gt;Exploring databricks SQL usage&lt;/p&gt;
&lt;p&gt;At my company, we adopted databricks SQL for most of our users. Some users have developed applications that use the JDBC connector, some users have built their dashboards, and some users write plain ad-hoc queries.&lt;/p&gt;
&lt;p&gt;We wanted to know what they queried, so we tried to use Unity Catalog&amp;rsquo;s insights, but it wasn&amp;rsquo;t enough for our case. We work with IOT and we are interested in what filters they apply within our tables.&lt;/p&gt;</description></item><item><title>Tweaking Spark Kafka</title><link>https://adrianabreu.com/blog/2023-10-27-tweaking-spark-kafka/</link><pubDate>Fri, 27 Oct 2023 12:06:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2023-10-27-tweaking-spark-kafka/</guid><description>&lt;p&gt;Well, I&amp;rsquo;m facing a huge interesting case. I&amp;rsquo;m working at Wallbox where we need to deal with billions of rows every day. Now we need to use Spark for some Kafka filtering and publish the results into different topics according to some rules.&lt;/p&gt;
&lt;p&gt;I won&amp;rsquo;t dig deep into the logic except for performance-related stuff, let&amp;rsquo;s try to increase the processing speed.&lt;/p&gt;
&lt;p&gt;When reading from Kafka you usually get 1 task per partition, so if you have 6 partitions and 48 cores you are not using 87.5 percent of your cluster. That could be adjusted with the following property &lt;code&gt;**minPartitions&lt;/code&gt;.**&lt;/p&gt;</description></item><item><title>KSQL, a horror tale</title><link>https://adrianabreu.com/blog/2023-10-22-ksql-a-horror-tale/</link><pubDate>Sat, 21 Oct 2023 22:52:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2023-10-22-ksql-a-horror-tale/</guid><description>&lt;p&gt;After spending several weeks working on a ksql solution to filter billions of events and determine their destination topic, I was disappointed to find that it did not live up to my expectations.&lt;/p&gt;
&lt;p&gt;I had hoped for a more robust product that would align with our needs. Previously, we utilized a similar filter in Spark, incurring traffic costs for both Confluent and AWS. With kSQL, the advantage was that we could avoid paying for AWS traffic.&lt;/p&gt;</description></item><item><title>Repairing metadata unity catalog</title><link>https://adrianabreu.com/blog/2023-10-02-repairing-metadata-unity-catalog/</link><pubDate>Mon, 02 Oct 2023 13:25:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2023-10-02-repairing-metadata-unity-catalog/</guid><description>&lt;p&gt;I&amp;rsquo;ve been subscribed to &lt;a href="https://www.dataengineeringweekly.com/p/data-engineering-weekly-148"&gt;https://www.dataengineeringweekly.com/p/data-engineering-weekly-148&lt;/a&gt; for years. This last number included several on-call posts on Medium. I found these quite useful.&lt;/p&gt;
&lt;p&gt;Today, I got an alert from Metaplane that a cost monitor dashboard was out of date. I checked the processes, and everything was fine. I ran a query to check the freshness of the data and it was ok too.&lt;/p&gt;
&lt;p&gt;Metaplane checks our delta table freshness by querying the table information available in the Unity Catalog. For some unknown reason that metadata didn&amp;rsquo;t receive any update. I ran an optimization operation (the table tiny) and the metadata didn&amp;rsquo;t update either.&lt;/p&gt;</description></item><item><title>Adding extra params on DatabricksRunNowOperator</title><link>https://adrianabreu.com/blog/2023-07-28-extra_params_databricksrunnow/</link><pubDate>Fri, 28 Jul 2023 16:00:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2023-07-28-extra_params_databricksrunnow/</guid><description>&lt;p&gt;With the &lt;a href="https://docs.databricks.com/api/workspace/jobs/runnow"&gt;new Databricks jobs API 2.1&lt;/a&gt; you have different parameters depending on the kind of tasks you have in your workflow. Like: jar_params, sql_params, python_params, notebook_params&amp;hellip;&lt;/p&gt;
&lt;p&gt;And not always the airflow operator is ready to handle all of the. If we check the &lt;a href="https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/run_now.html"&gt;current release of the DatabricksRunNowOperator&lt;/a&gt;, we can see that there is only support for:
notebook_params
python_params
python_named_parameters
jar_params
spark_submit_params
And not the query_params mentioned earlier. But there is a way of combining both, there is a param called &lt;em&gt;jsob&lt;/em&gt; that allows you to write the payload of a databricksrunnow and it will also merge the content of the JSON with your named_params!&lt;/p&gt;</description></item><item><title>Enabling Unity Catalog</title><link>https://adrianabreu.com/blog/2023-05-23-enabling-unity-catalog/</link><pubDate>Tue, 23 May 2023 07:48:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2023-05-23-enabling-unity-catalog/</guid><description>&lt;p&gt;I&amp;rsquo;ve spent the last few weeks setting up the unity catalog for my company. It&amp;rsquo;s been an extremely tiring process. And there are several concepts to bring here. My main point is to have a clear view of the requirements.&lt;/p&gt;
&lt;p&gt;Disclaimer: as of today with &lt;a href="https://github.com/databricks/terraform-provider-databricks"&gt;https://github.com/databricks/terraform-provider-databricks&lt;/a&gt; release 1.17.0, some steps should be done in an &amp;ldquo;awkward way&amp;rdquo; that is, the account API does not expose the catalog&amp;rsquo;s endpoint and should be done through a workspace.&lt;/p&gt;</description></item><item><title>Duplicates with delta, how can it be?</title><link>https://adrianabreu.com/blog/2023-03-20-delta-duplicates/</link><pubDate>Mon, 20 Mar 2023 09:50:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2023-03-20-delta-duplicates/</guid><description>&lt;p&gt;Long time without writing!
On highlights: I left my job at &lt;strong&gt;Schwarz It&lt;/strong&gt; in December last year, and now I&amp;rsquo;m a full-time employee at Wallbox! I&amp;rsquo;m really happy with my new job, and I&amp;rsquo;ve experienced interesting stuff. This one was just one of these strange cases where you start doubting the compiler.&lt;/p&gt;
&lt;h2 id="context"&gt;Context&lt;/h2&gt;
&lt;p&gt;One of my main tables represents sensor measures from our chargers with millisecond precision. The numbers are quite high, we are talking over 2 billion rows per day. So the analytic model doesn&amp;rsquo;t handle that level of granularity.
The analyst created a table that will make a window of 5 minutes, select some specific sensors and write there those values as a column. To keep the data consistent they were generating fake rows between sessions, so if a value was missing a synthetic value would be put in place.&lt;/p&gt;</description></item><item><title>Testing Databricks Photon</title><link>https://adrianabreu.com/blog/2022-08-12-testing-photon-engine/</link><pubDate>Fri, 12 Aug 2022 09:52:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2022-08-12-testing-photon-engine/</guid><description>&lt;p&gt;I was a bit skeptical about photon since I realized that it cost about double the amount of DBU, required specifically optimized machines and did not support UDFs (it was my main target).&lt;/p&gt;
&lt;p&gt;From the Databricks Official Docs:&lt;/p&gt;
&lt;h1 id="limitations"&gt;&lt;strong&gt;Limitations&lt;/strong&gt;&lt;/h1&gt;
&lt;ul&gt;
&lt;li&gt;Does not support Spark Structured Streaming.&lt;/li&gt;
&lt;li&gt;Does not support UDFs.&lt;/li&gt;
&lt;li&gt;Does not support RDD APIs.&lt;/li&gt;
&lt;li&gt;Not expected to improve short-running queries (&amp;lt;2 seconds), for example, queries against small amounts of data.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;a href="https://docs.databricks.com/runtime/photon.html"&gt;Photon runtime&lt;/a&gt;&lt;/p&gt;</description></item><item><title>Databricks Cluster Management</title><link>https://adrianabreu.com/blog/2022-07-30-databricks-cluster-management/</link><pubDate>Sat, 30 Jul 2022 13:52:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2022-07-30-databricks-cluster-management/</guid><description>&lt;p&gt;For the last few months, I&amp;rsquo;ve been into ETL optimization. Most of the changes were as dramatic as moving tables from ORC to delta revamping the partition strategy to some as simple as upgrading the runtime version to 10.4 so the ETL starts using low-shuffle merge.&lt;/p&gt;
&lt;p&gt;But at my job, we have a &lt;em&gt;lot&lt;/em&gt; of jobs. Each ETL can be easily launched at *30 with different parameters so I wanted to dig into the most effective strategy for it.&lt;/p&gt;</description></item><item><title>Pusing data to tinybird for free</title><link>https://adrianabreu.com/blog/2022-07-25-pushing-data-to-tinybird-free/</link><pubDate>Mon, 25 Jul 2022 07:28:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2022-07-25-pushing-data-to-tinybird-free/</guid><description>&lt;p&gt;So my azure subscription expired and I ended up losing the function I was using to feed my real-time data on analytics (part of the &lt;a href="https://github.com/adrianabreu/titsa-gtfs-api"&gt;Transportes Insulares de Tenerife SA&lt;/a&gt; analysis I was making).&lt;/p&gt;
&lt;p&gt;And after some struggle, I decided to move it to a GitHub action. Why? Because the free mins per month were more than enough and because I just needed some script to run on a cron and that script just makes a quest and a post. So, it was quite straightforward.&lt;/p&gt;</description></item><item><title>Associate Spark Developer Certification</title><link>https://adrianabreu.com/spark-certification/2022-07-21-passed-certification/</link><pubDate>Thu, 21 Jul 2022 12:16:32 +0000</pubDate><guid>https://adrianabreu.com/spark-certification/2022-07-21-passed-certification/</guid><description>&lt;p&gt;Yesterday I took (and passed with more than 90% yay!) the &lt;a href="https://databricks.com/learn/certification/apache-spark-developer-associate"&gt;Associate Spark Developer Certificaton&lt;/a&gt;. And before I forget I want to share my experience:&lt;/p&gt;
&lt;p&gt;In general:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;First of all, I needed to install Windows as there was no Linux support for the control software used during the exam.&lt;/li&gt;
&lt;li&gt;Secondly, you need to disable both the antivirus and the firewall before joining. I didn&amp;rsquo;t disable the antivirus and the technician contacted me as there was a problem with the webcam despite I was able to see myself.&lt;/li&gt;
&lt;li&gt;It is a controlled window started by the software, not a browser page (I had a good zoom on the example docs they provide, and well not the same on the software window).&lt;/li&gt;
&lt;li&gt;You can mark the questions for reviewing them later.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;About the exam:&lt;/p&gt;</description></item><item><title>Reading firebase data</title><link>https://adrianabreu.com/blog/2022-07-01-reading-firebase-data/</link><pubDate>Fri, 01 Jul 2022 07:28:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2022-07-01-reading-firebase-data/</guid><description>&lt;p&gt;Firebase is a common component nowadays for most mobile apps. And it can provide some useful insights, for example in my previous company we use it to detect where the people left at the initial app wizard. (We could measure it).&lt;/p&gt;
&lt;p&gt;It is quite simple to export your data to BigQuery: &lt;a href="https://firebase.google.com/docs/projects/bigquery-export"&gt;https://firebase.google.com/docs/projects/bigquery-export&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;But maybe your lake is in AWS or Azure. In the next lines, I will try to explain how to load the data in your lake and some improvements we have applied.&lt;/p&gt;</description></item><item><title>Qbeast</title><link>https://adrianabreu.com/blog/2022-06-30-qbeast/</link><pubDate>Thu, 30 Jun 2022 07:28:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2022-06-30-qbeast/</guid><description>&lt;p&gt;A few days ago I ran into &lt;a href="https://twitter.com/Qbeast_io"&gt;Qbeast&lt;/a&gt; which is an open-source project on top of delta lake I needed to dig into.&lt;/p&gt;
&lt;p&gt;This introductory post explains it quite well: &lt;a href="https://qbeast.io/qbeast-format-enhanced-data-lakehouse/"&gt;https://qbeast.io/qbeast-format-enhanced-data-lakehouse/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;The project is quite good and it seems helpful if you need to write your custom data source as everything is documented. And well as I&amp;rsquo;m in love with note-taking I want to dig into the following three topics:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Explaining how the format works (including optimizations)&lt;/li&gt;
&lt;li&gt;Describing how the sampling push is implementing&lt;/li&gt;
&lt;li&gt;Understanding the table tolerance&lt;/li&gt;
&lt;/ol&gt;
&lt;h1 id="1-qbeast-format"&gt;1. Qbeast format&lt;/h1&gt;
&lt;p&gt;This would be better explained with diagrams. Remember delta lake? We had a _delta_log folder with files pointing to files. Now Qbeast has extended this delta_log and has added some new properties.&lt;/p&gt;</description></item><item><title>Spark Dates</title><link>https://adrianabreu.com/spark-certification/2022-06-29-spark-dates/</link><pubDate>Wed, 29 Jun 2022 15:43:22 +0000</pubDate><guid>https://adrianabreu.com/spark-certification/2022-06-29-spark-dates/</guid><description>&lt;p&gt;I can perfectly describe this as the scariest part of the exam. I&amp;rsquo;m used to working with dates but I&amp;rsquo;m especially used to suffering from the typical UTC / not UTC / summer time hours difference.&lt;/p&gt;
&lt;p&gt;I will try to make some simple exercises for this, the idea would be:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;We have some sales data and god knows how the business people love to refresh super fast their dashboards on Databricks SQL. So we decided to aggregate at different levels the same KPI, our sales per store. Considering some data as:&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;data &lt;span style="color:#f92672"&gt;=&lt;/span&gt; [
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; (&lt;span style="color:#ae81ff"&gt;1656520076&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;1001&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;10&lt;/span&gt;),
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; (&lt;span style="color:#ae81ff"&gt;1656520321&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;1001&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;8&lt;/span&gt;),
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; (&lt;span style="color:#ae81ff"&gt;1656509025&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;1002&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;5&lt;/span&gt;),
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; (&lt;span style="color:#ae81ff"&gt;1656510826&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;1002&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;3&lt;/span&gt;),
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; (&lt;span style="color:#ae81ff"&gt;1656510056&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;1001&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;5&lt;/span&gt;),
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; (&lt;span style="color:#ae81ff"&gt;1656514076&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;1001&lt;/span&gt;, &lt;span style="color:#ae81ff"&gt;8&lt;/span&gt;),
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;]
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;ts &lt;span style="color:#f92672"&gt;=&lt;/span&gt; &lt;span style="color:#e6db74"&gt;&amp;#34;ts&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;store_id &lt;span style="color:#f92672"&gt;=&lt;/span&gt; &lt;span style="color:#e6db74"&gt;&amp;#34;store_id&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;amount &lt;span style="color:#f92672"&gt;=&lt;/span&gt; &lt;span style="color:#e6db74"&gt;&amp;#34;amount&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;df &lt;span style="color:#f92672"&gt;=&lt;/span&gt; spark&lt;span style="color:#f92672"&gt;.&lt;/span&gt;createDataFrame(data, [ts, store_id, amount])
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;We need to parse that data into a readable date as the first number is an epoch or &lt;em&gt;unix_time&lt;/em&gt;. Using the function from_unixttime this is quite simple:&lt;/p&gt;</description></item><item><title>Spark Cert Exam Practice</title><link>https://adrianabreu.com/spark-certification/2022-06-28-databricks-practice-exam/</link><pubDate>Tue, 28 Jun 2022 13:43:22 +0000</pubDate><guid>https://adrianabreu.com/spark-certification/2022-06-28-databricks-practice-exam/</guid><description>&lt;script 
src="https://cdn.jsdelivr.net/npm/quizdown@latest/public/build/quizdown.js"&gt;
&lt;/script&gt;
&lt;script 
 src="https://cdn.jsdelivr.net/npm/quizdown@latest/public/build/extensions/quizdownKatex.js"&gt;
&lt;/script&gt;
&lt;script 
 src="https://cdn.jsdelivr.net/npm/quizdown@latest/public/build/extensions/quizdownHighlight.js"&gt;
&lt;/script&gt;
&lt;script&gt;quizdown.register(quizdownHighlight).register(quizdownKatex).init()&lt;/script&gt; 

&lt;div class='quizdown'&gt;
 

---
primary_color: orange
secondary_color: lightgray
text_color: black
shuffle_questions: false
---

## Which of the following statements about the Spark driver is incorrect?

- [ ] The Spark driver is the node in which the Spark application's main method runs to ordinate the Spark application.
- [X] The Spark driver is horizontally scaled to increase overall processing throughput.
- [ ] The Spark driver contains the SparkContext object.
- [ ] The Spark driver is responsible for scheduling the execution of data by various worker nodes in cluster mode.
- [ ] The Spark driver should be as close as possible to worker nodes for optimal performance.

## Which of the following describes nodes in cluster-mode Spark?

- [ ] Nodes are the most granular level of execution in the Spark execution hierarchy.
- [ ] There is only one node and it hosts both the driver and executors.
- [ ] Nodes are another term for executors, so they are processing engine instances for performing computations.
- [ ] There are driver nodes and worker nodes, both of which can scale horizontally.
- [X] Worker nodes are machines that host the executors responsible for the execution of tasks

## Which of the following statements about slots is true?

- [ ] There must be more slots than executors.
- [ ] There must be more tasks than slots.
- [ ] Slots are the most granular level of execution in the Spark execution hierarchy.
- [ ] Slots are not used in cluster mode.
- [X] Slots are resources for parallelization within a Spark application.

## Which of the following is a combination of a block of data and a set of transformers that will run on a single executor?

- [ ] Executor
- [ ] Node
- [ ] Job
- [X] Task
- [ ] Slot

## Which of the following is a group of tasks that can be executed in parallel to compute the same set of operations on potentially multiple machines?

- [ ] Job
- [ ] Slot
- [ ] Executor
- [ ] Task
- [X] Stage

## Which of the following describes a shuffle?
- [X] A shuffle is the process by which data is compared across partitions.
- [ ] A shuffle is the process by which data is compared across executors.
- [ ] A shuffle is the process by which partitions are allocated to tasks.
- [ ] A shuffle is the process by which partitions are ordered for write.
- [ ] A shuffle is the process by which tasks are ordered for execution.

## DataFrame df is very large with a large number of partitions, more than there are executors in the cluster. Based on this situation, which of the following is incorrect? Assume there is one core per executor.

- [X] Performance will be suboptimal because not all executors will be utilized at the same time.
- [ ] Performance will be suboptimal because not all data can be processed at the same time.
- [ ] There will be a large number of shuffle connections performed on DataFrame df when operations inducing a shuffle are called.
- [ ] There will be a lot of overhead associated with managing resources for data processing within each task.
- [ ] There might be risk of out-of-memory errors depending on the size of the executors in the cluster.

## Which of the following operations will trigger evaluation?
- [ ] DataFrame.filter()
- [ ] DataFrame.distinct()
- [ ] DataFrame.intersect()
- [ ] DataFrame.join()
- [X] DataFrame.count()

## Which of the following describes the difference between transformations and actions?

- [ ] Transformations work on DataFrames/Datasets while actions are reserved for native language objects.
- [ ] There is no difference between actions and transformations.
- [ ] Actions are business logic operations that do not induce execution while transformations are execution triggers focused on returning results.
- [ ] Actions work on DataFrames/Datasets while transformations are reserved for native language objects.
- [X] Transformations are business logic operations that do not induce execution while actions are execution triggers focused on returning results.

## Which of the following DataFrame operations is always classified as a narrow transformation?
- [ ] DataFrame.sort()
- [ ] DataFrame.distinct()
- [ ] DataFrame.repartition()
- [X] DataFrame.select()
- [ ] DataFrame.join()

## Spark has a few different execution/deployment modes: cluster, client, and local. Which of the following describes Spark's execution/deployment mode? 

- [X] Spark's execution/deployment mode determines where the driver and executors are physically located when a Spark application is run
- [ ] Spark's execution/deployment mode determines which tasks are allocated to which executors in a cluster
- [ ] Spark's execution/deployment mode determines which node in a cluster of nodes is responsible for running the driver program
- [ ] Spark's execution/deployment mode determines exactly how many nodes the driver will connect to when a Spark application is run
- [ ] Spark's execution/deployment mode determines whether results are run interactively in a notebook environment or in batch

## Which of the following cluster configurations will ensure the completion of a Spark application in light of a worker node failure? Note: each configuration has roughly the same compute power using 100GB of RAM and 200 cores. 

![]("/images/databricks-cert/q12.png")

- [ ] Scenario #1
- [X] They should all ensure completion because worker nodes are fault-tolerant.
- [ ] Scenario #4
- [ ] Scenario #5
- [ ] Scenario #6

## Which of the following describes out-of-memory errors in Spark?

- [X] An out-of-memory error occurs when either the driver or an executor does not have enough memory to collect or process the data allocated to it.
- [ ] An out-of-memory error occurs when Spark's storage level is too lenient and allows data objects to be cached to both memory and disk.
- [ ] An out-of-memory error occurs when there are more tasks than are executors regardless of the number of worker nodes.
- [ ] An out-of-memory error occurs when the Spark application calls too many transformations in a row without calling an action regardless of the size of the data object on which the transformations are operating.
- [ ] An out-of-memory error occurs when too much data is allocated to the driver for computational purposes.

## Which of the following is the default storage level for persist() for a non-streaming DataFrame/Dataset?

- [X] MEMORY_AND_DISK
- [ ] MEMORY_AND_DISK_SER
- [ ] DISK_ONLY
- [ ] MEMORY_ONLY_SER
- [ ] MEMORY_ONLY

## Which of the following describes a broadcast variable?

- [ ] A broadcast variable is a Spark object that needs to be partitioned onto multiple worker nodes because it's too large to fit on a single worker node.
- [ ] A broadcast variable can only be created by an explicit call to the broadcast() operation.
- [ ] A broadcast variable is entirely cached on the driver node so it doesn't need to be present on any worker nodes.
- [X] A broadcast variable is entirely cached on each worker node so it doesn't need to be shipped or shuffled between nodes with each stage.
- [ ] A broadcast variable is saved to the disk of each worker node to be easily read into memory when needed.

## Which of the following operations is most likely to induce a skew in the size of your data's partitions?

- [ ] DataFrame.collect()
- [ ] DataFrame.cache()
- [ ] DataFrame.repartition(n)
- [X] DataFrame.coalesce(n)
- [ ] DataFrame.persist()

## Which of the following data structures are Spark DataFrames built on top of?
- [ ] Arrays
- [ ] Strings
- [X] RDDs
- [ ] Vectors
- [ ] SQL Tables

### Which of the following code blocks returns a DataFrame containing only column storeId and column **division** from DataFrame **storesDF?**

- [ ] `storesDF.select("storeId").select("division")`
- [ ] `storesDF.select(storeId, division)`
- [X] `storesDF.select("storeId", "division")`
- [ ] `storesDF.select(col("storeId", "division"))`
- [ ] `storesDF.select(storeId).select(division)`

## Which of the following code blocks returns a DataFrame containing all columns from DataFrame storesDF except for column sqft and column customerSatisfaction? A sample of DataFrame storesDF is below:
![]("/images/databricks-cert/q19.png")

- [X] `storesDF.drop("sqft", "customerSatisfaction")`
- [ ] `storesDF.select("storeId", "open", "openDate", "division")`
- [ ] `storesDF.select(-col(sqft), -col(customerSatisfaction))`
- [ ] `storesDF.drop(sqft, customerSatisfaction)`
- [ ] `storesDF.drop(col(sqft), col(customerSatisfaction))`

## The below code shown block contains an error. The code block is intended to return a DataFrame containing only the rows from DataFrame storesDF where the value in DataFrame storesDF's "sqft" column is less than or equal to 25,000. Assume DataFrame storesDF is the only defined language variable. Identify the error. Code block: `storesDF.filter(sqft &lt;= 25000)`

- [ ] The column name **sqft** needs to be quoted like s**toresDF.filter("sqft" &lt;=000).**
- [X] The column name sqft needs to be quoted and wrapped in the **col()** function like **storesDF.filter(col("sqft") &lt;= 25000).**
- [ ] The sign in the logical condition inside **filter()** needs to be changed from &lt;= to &gt;.
- [ ] The sign in the logical condition inside **filter()** needs to be changed from &lt;= to &gt;=.
- [ ] The column name sqft needs to be wrapped in the **col()** function like **storesDF.filter(col(sqft) &lt;= 25000).**

## The code block shown below should return a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: storesDF.__**1__**(__**2**__ __3__ __4__)

- [X] ```
 1. filter
 2. (col("sqft") &lt;= 25000)
 3. |
 4. (col("customerSatisfaction") &gt;= 30)
 ```
- [ ] ```
 1. drop
 2. (col(sqft) &lt;= 25000)
 3. |
 4. (col(customerSatisfaction) &gt;= 30)
```
- [ ] ```
 1. filter
 2. col("sqft") &lt;= 25000
 3. |
 4. col("customerSatisfaction") &gt;= 30
```
- [ ] ```
 1. filter
 2. col("sqft") &lt;= 25000
 3. or
 4. col("customerSatisfaction") &gt;= 30
```
- [ ] ```
 1. filter
 2. (col("sqft") &lt;= 25000)
 3. or
 4. (col("customerSatisfaction") &gt;= 30)
```

## Which of the following operations can be used to convert a DataFrame column from one type to another type?

- [X] col().cast()
- [ ] convert()
- [ ] castAs()
- [ ] col().coerce()
- [ ] col()

## Which of the following code blocks returns a new DataFrame with a new column sqft100 that is 1/100th of column sqft in DataFrame storesDF? Note that column sqft100 is not in the original DataFrame storesDF.

- [ ] `storesDF.withColumn("sqft100", col("sqft") * 100)`
- [ ] `storesDF.withColumn("sqft100", sqft / 100)`
- [ ] `storesDF.withColumn(col("sqft100"), col("sqft") / 100)`
- [X] `storesDF.withColumn("sqft100", col("sqft") / 100)`
- [ ] `storesDF.newColumn("sqft100", sqft / 100)`

## Which of the following code blocks returns a new DataFrame from DataFrame storesDF where column numberOfManagers is the constant integer 1?

- [ ] `storesDF.withColumn("numberOfManagers", col(1))`
- [ ] `storesDF.withColumn("numberOfManagers", 1)`
- [X] `storesDF.withColumn("numberOfManagers", lit(1))`
- [ ] `storesDF.withColumn("numberOfManagers", lit("1"))`
- [ ] `storesDF.withColumn("numberOfManagers", IntegerType(1))`

## The code block shown below contains an error. The code block intends to return a new DataFrame where column storeCategory from DataFrame storesDF is split at the underscore character into column storeValueCategory and column storeSizeCategory. Identify the error. A sample of DataFrame storesDF is displayed below: Code block:
```
(storesDF.withColumn(
 "storeValueCategory", col("storeCategory").split("*")[0]
 ).withColumn(
 "storeSizeCategory", col("storeCategory").split("*")[1]
 )
)
```
![]("/images/spark-certification/q25.png")

- [ ] The split() operation comes from the imported functions object. It accepts a string column name and split character as arguments. It is not a method of a Column object.
- [X] The split() operation comes from the imported functions object. It accepts a Column object and split character as arguments. It is not a method of a Column object.
- [ ] The index values of 0 and 1 should be provided as second arguments to the split() operation rather than indexing the result.
- [ ] The index values of 0 and 1 are not correct — they should be 1 and 2, respectively.
- [ ] The withColumn() operation cannot be called twice in a row.


## Which of the following operations can be used to split an array column into an individual DataFrame row for each element in the array?
- [ ] extract()
- [ ] split()
- [X] explode()
- [ ] arrays_zip()
- [ ] unpack()

## Which of the following code blocks returns a new DataFrame where column storeCategory is an all-lowercase version of column storeCategory in DataFrame storesDF? Assume DataFrame storesDF is the only defined language variable.

- [X] `storesDF.withColumn("storeCategory", lower(col("storeCategory")))`
- [ ] `storesDF.withColumn("storeCategory", coll("storeCategory").lower())`
- [ ] `storesDF.withColumn("storeCategory", tolower(col("storeCategory")))`
- [ ] `storesDF.withColumn("storeCategory", lower("storeCategory"))`
- [ ] `storesDF.withColumn("storeCategory", lower(storeCategory))`

## The code block shown below contains an error. The code block is intended to return a new DataFrame where column division from DataFrame storesDF has been renamed to column state and column managerName from DataFrame storesDF has been renamed to column managerFullName. Identify the error. Code block:
```
(storesDF.withColumnRenamed("state", "division")
.withColumnRenamed("managerFullName", "managerName"))
```

- [ ] Both arguments to operation withColumnRenamed() should be wrapped in the col() operation.
- [ ] The operations withColumnRenamed() should not be called twice, and the first argument should be ["state", "division"] and the second argument should be["managerFullName", "managerName"].
- [ ] The old columns need to be explicitly dropped.
- [X] The first argument to operation withColumnRenamed() should be the old column name and the second argument should be the new column name.
- [ ] The operation withColumnRenamed() should be replaced with withColumn().

## Which of the following code blocks returns a DataFrame where rows in DataFrame storesDF containing missing values in every column have been dropped?

- [ ] storesDF.nadrop("all")
- [ ] storesDF.na.drop("all", subset = "sqft")
- [ ] storesDF.dropna()
- [ ] storesDF.na.drop()
- [X] storesDF.na.drop("all")

## Which of the following operations fails to return a DataFrame where every row is unique?

- [ ] DataFrame.distinct()
- [ ] DataFrame.drop_duplicates(subset = None)
- [ ] DataFrame.drop_duplicates()
- [ ] DataFrame.dropDuplicates()
- [X] DataFrame.drop_duplicates(subset = "all")

## Which of the following code blocks will not always return the exact number of distinct values in column division?

- [X] `storesDF.agg(approx_count_distinct(col("division")).alias("divisionDistinct"))`
- [ ] `storesDF.agg(approx_count_distinct(col("division"), 0).alias("divisionDistinct"))`
- [ ] `storesDF.agg(countDistinct(col("division")).alias("divisionDistinct"))`
- [ ] `storesDF.select("division").dropDuplicates().count()`
- [ ] `storesDF.select("division").distinct().count()`

## The code block shown below should return a new DataFrame with the mean of column sqft from DataFrame storesDF in column sqftMean. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block:
`storesDF.__**1__**(__2__(__**3**__).alias("sqftMean"))`

- [X] ```
 1. agg
 2. mean
 3. col("sqft")
```
- [ ] ```
 1. mean
 2. col
 3. "sqft"
```
- [ ] ```
 1. withColumn
 2. mean
 3. col("sqft")
```
- [ ] ```
 1. agg
 2. mean
 3. "sqft"
```
- [ ] ```
 1. agg
 2. average
 3. col("sqft")
```

## Which of the following code blocks returns the number of rows in DataFrame storesDF?

- [ ] storesDF.withColumn("numberOfRows", count())
- [ ] storesDF.withColumn(count().alias("numberOfRows"))
- [ ] storesDF.countDistinct()
- [X] storesDF.count()
- [ ] storesDF.agg(count())

## Which of the following code blocks returns the sum of the values in column sqft in DataFrame storesDF grouped by distinct value in column division?

- [ ] storesDF.groupBy.agg(sum(col("sqft")))
- [ ] storesDF.groupBy("division").agg(sum())
- [ ] storesDF.agg(groupBy("division").sum(col("sqft")))
- [ ] storesDF.groupby.agg(sum(col("sqft")))
- [X] storesDF.groupBy("division").agg(sum(col("sqft")))

## Which of the following code blocks returns a DataFrame containing summary statistics only for column sqft in DataFrame storesDF?

- [ ] storesDF.summary("mean")
- [X] storesDF.describe("sqft")
- [ ] storesDF.summary(col("sqft"))
- [ ] storesDF.describeColumn("sqft")
- [ ] storesDF.summary()

## Which of the following operations can be used to sort the rows of a DataFrame?

- [X] sort() and orderBy()
- [ ] orderby()
- [ ] sort() and orderby()
- [ ] orderBy()
- [ ] sort()

## The code block shown below contains an error. The code block is intended to return a 15 percent sample of rows from DataFrame storesDF without replacement. Identify the error. Code block: storesDF.sample(True, fraction = 0.15)

- [ ] There is no argument specified to the seed parameter.
- [ ] There is no argument specified to the withReplacement parameter.
- [ ] The sample() operation does not sample without replacement — sampleby() should be used instead.
- [ ] The sample() operation is not reproducible.
- [X] The first argument True sets the sampling to be with replacement.

## Which of the following operations can be used to return the top n rows from a DataFrame?

- [ ] DataFrame.n()
- [X] DataFrame.take(n)
- [ ] DataFrame.head
- [ ] DataFrame.show(n)
- [ ] DataFrame.collect(n)

## The code block shown below should extract the value for column sqft from the first row of DataFrame storesDF. Choose the response that correctly fills in the numbered blanks within the code block to complete this task.
Code block:
`__**1__.__2 __.__3 __**`

- [ ] ```
 1. storesDF
 2. first
 3. col("sqft")
```
- [ ] ```
 1. storesDF
 2. first
 3. sqft
```
- [ ] ```
 1. storesDF
 2. first
 3. ["sqft"]
```
- [X] ```
 1. storesDF
 2. first()
 3. sqft
```
- [ ] ```
 1. storesDF
 2. first()
 3. col("sqft")
```

## Which of the following lines of code prints the schema of a DataFrame?

- [ ] print(storesDF)
- [ ] storesDF.schema
- [ ] print(storesDF.schema())
- [X] DataFrame.printSchema()
- [ ] DataFrame.schema()

## In what order should the below lines of code be run in order to create and register a SQL UDF named "ASSESS_PERFORMANCE" using the Python function assessPerformance and apply it to column customerSatistfaction in table stores?
```
Lines of code:
1.`spark.udf.register("ASSESS_PERFORMANCE", assessPerformance)`
2.`spark.sql("SELECT customerSatisfaction, assessPerformance(customerSatisfaction) AS result FROM stores")`
3.`spark.udf.register(assessPerformance, "ASSESS_PERFORMANCE")`
4. `spark.sql("SELECT customerSatisfaction, ASSESS_PERFORMANCE(customerSatisfaction) AS result FROM
stores")`
```

- [ ] 3, 4
- [X] 1, 4
- [ ] 3, 2
- [ ] 2
- [ ] 1, 2

## In what order should the below lines of code be run in order to create a Python UDF assessPerformanceUDF() using the integer-returning Python function assessPerformance and apply it to column customerSatisfaction in DataFrame storesDF?
```
Lines of code:
1. assessPerformanceUDF = udf(assessPerformance, IntegerType)
2. assessPerformanceUDF = spark.register.udf("ASSESS_PERFORMANCE",
assessPerformance)
3. assessPerformanceUDF = udf(assessPerformance, IntegerType())
4. storesDF.withColumn("result",
assessPerformanceUDF(col("customerSatisfaction")))
5. storesDF.withColumn("result",
assessPerformance(col("customerSatisfaction")))
6. storesDF.withColumn("result",
ASSESS_PERFORMANCE(col("customerSatisfaction")))
```

- [X] 3, 4
- [ ] 2, 6
- [ ] 3, 5
- [ ] 1, 4
- [ ] 2, 5

## Which of the following operations can execute a SQL query on a table?

- [ ] spark.query()
- [ ] DataFrame.sql()
- [X] spark.sql()
- [ ] DataFrame.createOrReplaceTempView()
- [ ] DataFrame.createTempView()

## Which of the following code blocks creates a single-column DataFrame from Python list years which is made up of integers?

- [ ] `spark.createDataFrame([years], IntegerType())`
- [X] `spark.createDataFrame(years, IntegerType())`
- [ ] `spark.DataFrame(years, IntegerType())`
- [ ] `spark.createDataFrame(years)`
- [ ] `spark.createDataFrame(years, IntegerType)`


## Which of the following operations can be used to cache a DataFrame only in Spark’s memory assuming the default arguments can be updated?

- [ ] DataFrame.clearCache()
- [ ] DataFrame.storageLevel
- [ ] StorageLevel
- [X] DataFrame.persist()
- [ ] DataFrame.cache()

## The code block shown below contains an error. The code block is intended to return a new 4-partition DataFrame from the 8-partition DataFrame storesDF without inducing a shuffle. Identify the error. Code block: storesDF.repartition(4)

- [ ] The repartition operation will only work if the DataFrame has been cached to memory. 
- [ ] The repartition operation requires a column on which to partition rather than a number of partitions.
- [ ] The number of resulting partitions, 4, is not achievable for an 8-partition DataFrame.
- [X] The repartition operation induced a full shuffle. The coalesce operation should be used instead.
- [ ] The repartition operation cannot guarantee the number of result partitions.

## Which of the following code blocks will always return a new 12-partition DataFrame from the 8-partition DataFrame storesDF?

- [ ] storesDF.coalesce(12)
- [ ] storesDF.repartition()
- [X] storesDF.repartition(12)
- [ ] storesDF.coalesce()
- [ ] storesDF.coalesce(12, "storeId")

## Which of the following Spark config properties represents the number of partitions used in wide transformations like join()?

- [x] `spark.sql.shuffle.partitions`
- [ ] `spark.shuffle.partitions`
- [ ] `spark.shuffle.io.maxRetries`
- [ ] `spark.shuffle.file.buffer`
- [ ] `spark.default.parallelism`

### In what order should the below lines of code be run in order to return a DataFrame containing a column openDateString, a string representation of Java’s SimpleDateFormat? Note that column openDate is of type integer and represents a date in the UNIX epoch format — the number of seconds since midnight on January 1st, 1970. An example of Java's SimpleDateFormat is "Sunday, Dec 4, 2008 1:05 PM". A sample of storesDF is displayed below:
```
Lines of code:
1. `storesDF.withColumn("openDateString",
from_unixtime(col("openDate"), simpleDateFormat))`
2. `simpleDateFormat = "EEEE, MMM d, yyyy h:mm a"`
3. `storesDF.withColumn("openDateString",
from_unixtime(col("openDate"), SimpleDateFormat()))`
4.`storesDF.withColumn("openDateString",
date_format(col("openDate"), simpleDateFormat))`
5.`storesDF.withColumn("openDateString",
date_format(col("openDate"), SimpleDateFormat()))`
6.`simpleDateFormat = "wd, MMM d, yyyy h:mm a"`
```

- [ ] 2, 3
- [X] 2, 1
- [ ] 6, 5
- [ ] 2, 4
- [ ] 6, 1

## Which of the following code blocks returns a DataFrame containing a column month, an integer representation of the month from column openDate from DataFrame storesDF? Note that column openDate is of type integer and represents a date in the UNIX epoch format — the number of seconds since midnight on January 1st, 1970. A sample of storesDF is displayed below:
![]("/images/databricks-cert/q50.png")

- [ ] `storesDF.withColumn("month", getMonth(col("openDate")))`
- [X] `storesDF.withColumn("openTimestamp", col("openDate").cast("Timestamp")).withColumn("month", month(col("openTimestamp")))`
- [ ] `storesDF.withColumn("openDateFormat", col("openDate").cast("Date")).withColumn("month", month(col("openDateFormat")))`
- [ ] `storesDF.withColumn("month", substr(col("openDate"), 4, 2))`
- [ ] `storesDF.withColumn("month", month(col("openDate")))`


## Which of the following operations performs an inner join on two DataFrames?

- [ ] DataFrame.innerJoin()
- [X] DataFrame.join()
- [ ] Standalone join() function
- [ ] DataFrame.merge()
- [ ] DataFrame.crossJoin()

## Which of the following code blocks returns a new DataFrame that is the result of an outer join between DataFrame storesDF and DataFrame employeesDF on column storeId?

- [X] storesDF.join(employeesDF, "storeId", "outer")
- [ ] storesDF.join(employeesDF, "storeId")
- [ ] storesDF.join(employeesDF, "outer", col("storeId"))
- [ ] storesDF.join(employeesDF, "outer", storesDF.storeId == employeesDF.storeId)
- [ ] storesDF.merge(employeesDF, "outer", col("storeId"))

## The below code block contains an error. The code block is intended to return a new DataFrame that is the result of an inner join between DataFrame storesDF and DataFrame employeesDF on column storeId and column employeeId which are in both DataFrames. Identify the error.
Code block:
`storesDF.join(employeesDF, [col("storeId"), col("employeeId")])`

- [ ] The join() operation is a standalone function rather than a method of DataFrame — the join() operation should be called where its first two arguments are storesDF and employeesDF.
- [ ] There must be a third argument to join() because the default to the how parameter is not "inner".
- [ ] The col("storeId") and col("employeeId") arguments should not be separate elements of a list — they should be tested to see if they're equal to one another like col("storeId") == col("employeeId").
- [ ] There is no DataFrame.join() operation — DataFrame.merge() should be used instead.
- [X] The references to "storeId" and "employeeId" should not be inside the col() function — removing the col() function should result in a successful join.

## Which of the following Spark properties is used to configure the broadcasting of a DataFrame without the use of the broadcast() operation?

- [X] spark.sql.autoBroadcastJoinThreshold
- [ ] spark.sql.broadcastTimeout
- [ ] spark.broadcast.blockSize
- [ ] spark.broadcast.compress
- [ ] spark.executor.memoryOverhead


## The code block shown below should return a new DataFrame that is the result of a cross join between DataFrame storesDF and DataFrame employeesDF. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block:

__***1*__**.__**2__**(__3__)

- [ ] ```
 1. storesDF
 2. crossJoin
 3. employeesDF, "storeId"
```
- [ ] ```
 1. storesDF
 2. join
 3. employeesDF, "cross"
```
- [ ] ```
 1. storesDF
 2. crossJoin
 3. employeesDF, "storeId"
```
- [ ] ```
 1. storesDF
 2. join
 3. employeesDF, "storeId", "cross"
```
- [X] ```
 1. storesDF
 2. crossJoin
 3. employeesDF
```

## Which of the following operations performs a position-wise union on two DataFrames?

- [ ] The standalone concat() function
- [ ] The standalone unionAll() function
- [ ] The standalone union() function
- [ ] DataFrame.unionByName()
- [X] DataFrame.union()

## Which of the following code blocks writes DataFrame storesDF to file path filePath as parquet?

- [ ] storesDF.write.option("parquet").path(filePath)
- [ ] storesDF.write.path(filePath)
- [ ] storesDF.write().parquet(filePath)
- [ ] storesDF.write(filePath)
- [X] storesDF.write.parquet(filePath)

## The code block shown below contains an error. The code block is intended to write DataFrame storesDF to file path filePath as parquet and partition by values in column division. Identify the error. Code block:

`storesDF.write.repartition("division").parquet(filePath)`

- [ ] The argument division to operation repartition() should be wrapped in the col() function to return a Column object.
- [ ] There is no parquet() operation for DataFrameWriter — the save() operation should be used instead.
- [X] There is no repartition() operation for DataFrameWriter — the partitionBy() operation should be used instead.
- [ ] DataFrame.write is an operation — it should be followed by parentheses to return a
DataFrameWriter.
- [ ] The mode() operation must be called to specify that this write should not overwrite
existing files.

## Which of the following code blocks reads a parquet at the file path filePath into a DataFrame?

- [ ] spark.read().parquet(filePath)
- [ ] spark.read().path(filePath, source = "parquet")
- [ ] spark.read.path(filePath, source = "parquet")
- [x] spark.read.parquet(filePath)
- [ ] spark.read().path(filePath)


## Which of the following code blocks reads JSON at the file path filePath into a DataFrame with the specified schema schema?

- [ ] spark.read().schema(schema).format(json).load(filePath)
- [ ] spark.read().schema(schema).format("json").load(filePath)
- [ ] spark.read.schema("schema").format("json").load(filePath)
- [ ] spark.read.schema("schema").format("json").load(filePath)
- [x] spark.read.schema(schema).format("json").load(filePath)



&lt;/div&gt;</description></item><item><title>Spark User Defined Functions</title><link>https://adrianabreu.com/spark-certification/2022-06-19-spark-udf-udaf/</link><pubDate>Sun, 19 Jun 2022 14:43:22 +0000</pubDate><guid>https://adrianabreu.com/spark-certification/2022-06-19-spark-udf-udaf/</guid><description>&lt;p&gt;Sometimes we need to execute arbitrary Scala code on Spark. We may need to use an external library or so on. For that, we have the UDF, which accepts and return one or more columns.&lt;/p&gt;
&lt;p&gt;When we have a function we need to register it on Spark so we can use it on our worker machines. If you are using Scala or Java, the udf can run inside the Java Virtual Machine so there&amp;rsquo;s a little extra penalty. But from Python, there is an extra penalty as Spark needs to start a Python process on the worker, serialize the data from JVM to Python, run the function and then serialize the result to the JVM.&lt;/p&gt;</description></item><item><title>Spark DataSources</title><link>https://adrianabreu.com/spark-certification/2022-06-11-spark-data-sources/</link><pubDate>Sat, 11 Jun 2022 16:43:22 +0000</pubDate><guid>https://adrianabreu.com/spark-certification/2022-06-11-spark-data-sources/</guid><description>&lt;p&gt;As estated in the &lt;a href="https://adrianabreu.com/spark-certification/2022-06-10-spark-structured-api"&gt;structured api section&lt;/a&gt;, Spark supports a lot of sources with a lot of options. There is no other goal for this post than to clarify how the most common ones work and how they will be converted to &lt;strong&gt;DataFrames&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;First, all the supported sources are listed here: &lt;a href="https://spark.apache.org/docs/latest/sql-data-sources.html"&gt;https://spark.apache.org/docs/latest/sql-data-sources.html&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;And we can focus on the typical ones: JSON, CSV and Parquet (as those are the typical format on open-source data).&lt;/p&gt;</description></item><item><title>Spark Dataframes</title><link>https://adrianabreu.com/spark-certification/2022-06-10-spark-structured-api/</link><pubDate>Fri, 10 Jun 2022 17:02:32 +0000</pubDate><guid>https://adrianabreu.com/spark-certification/2022-06-10-spark-structured-api/</guid><description>&lt;p&gt;Spark was initially released for dealing with a particular type of data called &lt;strong&gt;RDD&lt;/strong&gt;. Nowadays we work with abstract structures on top of it, and the following tables summarize them.&lt;/p&gt;
&lt;table&gt;
 &lt;thead&gt;
 &lt;tr&gt;
 &lt;th&gt;Type&lt;/th&gt;
 &lt;th&gt;Description&lt;/th&gt;
 &lt;th&gt;Advantages&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody&gt;
 &lt;tr&gt;
 &lt;td&gt;Datasets&lt;/td&gt;
 &lt;td&gt;Structured composed of a list of &lt;T&gt; where you can specify your custom class (only Scala)&lt;/td&gt;
 &lt;td&gt;Type-safe operations, support for operations that cannot be expressed otherwise.&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Dataframes&lt;/td&gt;
 &lt;td&gt;Datasets of type Row (a generic spark type)&lt;/td&gt;
 &lt;td&gt;Allow optimizations and are more flexible&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;SQL tables and views&lt;/td&gt;
 &lt;td&gt;Same as Dataframes but in the scope of databases instead of programming languages&lt;/td&gt;
 &lt;td&gt;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Let&amp;rsquo;s dig into the Dataframes.
They are a data abstraction for interacting with name columns, those names are defined in a &lt;strong&gt;schema&lt;/strong&gt;.&lt;/p&gt;</description></item><item><title>Spark Execution</title><link>https://adrianabreu.com/spark-certification/2022-06-08-spark-execution/</link><pubDate>Wed, 08 Jun 2022 17:02:32 +0000</pubDate><guid>https://adrianabreu.com/spark-certification/2022-06-08-spark-execution/</guid><description>&lt;p&gt;Spark provides an api and an engine, that engine is responsible for analyzing the code and performing several optimizations. But how does this work?
We can do two kinds of operations with Spark, transformations and actions.&lt;/p&gt;
&lt;p&gt;Transformations are operations on top of the data that modify the data but do not yield a result directly, that is because they all are lazily evaluated so, you can add new columns, filter rows, or perform some computations that won&amp;rsquo;t be executed immediately.&lt;/p&gt;</description></item><item><title>Spark Architecture</title><link>https://adrianabreu.com/spark-certification/2022-06-07-spark-architecture/</link><pubDate>Tue, 07 Jun 2022 17:02:32 +0000</pubDate><guid>https://adrianabreu.com/spark-certification/2022-06-07-spark-architecture/</guid><description>&lt;p&gt;Spark works on top of a cluster supervised by a cluster manager. The later is responsible of:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Tracking resource allocation across all applications running on the cluster.&lt;/li&gt;
&lt;li&gt;Monitoring the health of all the nodes.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Inside each node there is a node manager which is responsible to track each node health and resources and inform the cluster manager.&lt;/p&gt;



&lt;div class="goat svg-container "&gt;
 
 &lt;svg
 xmlns="http://www.w3.org/2000/svg"
 font-family="Menlo,Lucida Console,monospace"
 
 viewBox="0 0 328 105"
 &gt;
 &lt;g transform='translate(8,16)'&gt;
&lt;path d='M 0,16 L 136,16' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 176,16 L 208,16' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 136,32 L 176,32' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 0,48 L 136,48' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 176,48 L 208,48' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 176,80 L 208,80' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 0,16 L 0,48' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 136,16 L 136,32' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 136,32 L 136,48' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 176,16 L 176,32' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 176,32 L 176,48' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 176,48 L 176,64' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;path d='M 176,64 L 176,80' fill='none' stroke='currentColor'&gt;&lt;/path&gt;
&lt;text text-anchor='middle' x='8' y='36' fill='currentColor' style='font-size:1em'&gt;C&lt;/text&gt;
&lt;text text-anchor='middle' x='16' y='36' fill='currentColor' style='font-size:1em'&gt;l&lt;/text&gt;
&lt;text text-anchor='middle' x='24' y='36' fill='currentColor' style='font-size:1em'&gt;u&lt;/text&gt;
&lt;text text-anchor='middle' x='32' y='36' fill='currentColor' style='font-size:1em'&gt;s&lt;/text&gt;
&lt;text text-anchor='middle' x='40' y='36' fill='currentColor' style='font-size:1em'&gt;t&lt;/text&gt;
&lt;text text-anchor='middle' x='48' y='36' fill='currentColor' style='font-size:1em'&gt;e&lt;/text&gt;
&lt;text text-anchor='middle' x='56' y='36' fill='currentColor' style='font-size:1em'&gt;r&lt;/text&gt;
&lt;text text-anchor='middle' x='72' y='36' fill='currentColor' style='font-size:1em'&gt;M&lt;/text&gt;
&lt;text text-anchor='middle' x='80' y='36' fill='currentColor' style='font-size:1em'&gt;a&lt;/text&gt;
&lt;text text-anchor='middle' x='88' y='36' fill='currentColor' style='font-size:1em'&gt;n&lt;/text&gt;
&lt;text text-anchor='middle' x='96' y='36' fill='currentColor' style='font-size:1em'&gt;a&lt;/text&gt;
&lt;text text-anchor='middle' x='104' y='36' fill='currentColor' style='font-size:1em'&gt;g&lt;/text&gt;
&lt;text text-anchor='middle' x='112' y='36' fill='currentColor' style='font-size:1em'&gt;e&lt;/text&gt;
&lt;text text-anchor='middle' x='120' y='36' fill='currentColor' style='font-size:1em'&gt;r&lt;/text&gt;
&lt;text text-anchor='middle' x='224' y='20' fill='currentColor' style='font-size:1em'&gt;N&lt;/text&gt;
&lt;text text-anchor='middle' x='224' y='52' fill='currentColor' style='font-size:1em'&gt;N&lt;/text&gt;
&lt;text text-anchor='middle' x='224' y='84' fill='currentColor' style='font-size:1em'&gt;N&lt;/text&gt;
&lt;text text-anchor='middle' x='232' y='20' fill='currentColor' style='font-size:1em'&gt;o&lt;/text&gt;
&lt;text text-anchor='middle' x='232' y='52' fill='currentColor' style='font-size:1em'&gt;o&lt;/text&gt;
&lt;text text-anchor='middle' x='232' y='84' fill='currentColor' style='font-size:1em'&gt;o&lt;/text&gt;
&lt;text text-anchor='middle' x='240' y='20' fill='currentColor' style='font-size:1em'&gt;d&lt;/text&gt;
&lt;text text-anchor='middle' x='240' y='52' fill='currentColor' style='font-size:1em'&gt;d&lt;/text&gt;
&lt;text text-anchor='middle' x='240' y='84' fill='currentColor' style='font-size:1em'&gt;d&lt;/text&gt;
&lt;text text-anchor='middle' x='248' y='20' fill='currentColor' style='font-size:1em'&gt;e&lt;/text&gt;
&lt;text text-anchor='middle' x='248' y='52' fill='currentColor' style='font-size:1em'&gt;e&lt;/text&gt;
&lt;text text-anchor='middle' x='248' y='84' fill='currentColor' style='font-size:1em'&gt;e&lt;/text&gt;
&lt;text text-anchor='middle' x='264' y='20' fill='currentColor' style='font-size:1em'&gt;M&lt;/text&gt;
&lt;text text-anchor='middle' x='264' y='52' fill='currentColor' style='font-size:1em'&gt;M&lt;/text&gt;
&lt;text text-anchor='middle' x='264' y='84' fill='currentColor' style='font-size:1em'&gt;M&lt;/text&gt;
&lt;text text-anchor='middle' x='272' y='20' fill='currentColor' style='font-size:1em'&gt;a&lt;/text&gt;
&lt;text text-anchor='middle' x='272' y='52' fill='currentColor' style='font-size:1em'&gt;a&lt;/text&gt;
&lt;text text-anchor='middle' x='272' y='84' fill='currentColor' style='font-size:1em'&gt;a&lt;/text&gt;
&lt;text text-anchor='middle' x='280' y='20' fill='currentColor' style='font-size:1em'&gt;n&lt;/text&gt;
&lt;text text-anchor='middle' x='280' y='52' fill='currentColor' style='font-size:1em'&gt;n&lt;/text&gt;
&lt;text text-anchor='middle' x='280' y='84' fill='currentColor' style='font-size:1em'&gt;n&lt;/text&gt;
&lt;text text-anchor='middle' x='288' y='20' fill='currentColor' style='font-size:1em'&gt;a&lt;/text&gt;
&lt;text text-anchor='middle' x='288' y='52' fill='currentColor' style='font-size:1em'&gt;a&lt;/text&gt;
&lt;text text-anchor='middle' x='288' y='84' fill='currentColor' style='font-size:1em'&gt;a&lt;/text&gt;
&lt;text text-anchor='middle' x='296' y='20' fill='currentColor' style='font-size:1em'&gt;g&lt;/text&gt;
&lt;text text-anchor='middle' x='296' y='52' fill='currentColor' style='font-size:1em'&gt;g&lt;/text&gt;
&lt;text text-anchor='middle' x='296' y='84' fill='currentColor' style='font-size:1em'&gt;g&lt;/text&gt;
&lt;text text-anchor='middle' x='304' y='20' fill='currentColor' style='font-size:1em'&gt;e&lt;/text&gt;
&lt;text text-anchor='middle' x='304' y='52' fill='currentColor' style='font-size:1em'&gt;e&lt;/text&gt;
&lt;text text-anchor='middle' x='304' y='84' fill='currentColor' style='font-size:1em'&gt;e&lt;/text&gt;
&lt;text text-anchor='middle' x='312' y='20' fill='currentColor' style='font-size:1em'&gt;r&lt;/text&gt;
&lt;text text-anchor='middle' x='312' y='52' fill='currentColor' style='font-size:1em'&gt;r&lt;/text&gt;
&lt;text text-anchor='middle' x='312' y='84' fill='currentColor' style='font-size:1em'&gt;r&lt;/text&gt;
&lt;/g&gt;

 &lt;/svg&gt;
 
&lt;/div&gt;
&lt;p&gt;When we run a Spark application we generate processes inside the cluster where one node will act as a Driver and the rest will be Workers. Here there are two main points:&lt;/p&gt;</description></item><item><title>Faker with PySpark</title><link>https://adrianabreu.com/blog/2022-05-31-faker-pyspark/</link><pubDate>Tue, 31 May 2022 09:28:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2022-05-31-faker-pyspark/</guid><description>&lt;p&gt;I’m preparing a small blog post about some tweakings I’ve done for a delta table, but I want to dig into the Spark UI differences before this. As this was done as part of my work I’m reproducing the problem with some generated data.&lt;/p&gt;
&lt;p&gt;I didn’t know about &lt;a href="https://faker.readthedocs.io/en/master/"&gt;Faker&lt;/a&gt; and &lt;em&gt;boy&lt;/em&gt; it is really simple and easy.&lt;/p&gt;
&lt;p&gt;In this case, I want to generate a small dataset for a dimension product table including its id, category and price.&lt;/p&gt;</description></item><item><title>Git 101</title><link>https://adrianabreu.com/blog/2022-03-21-git-intro/</link><pubDate>Mon, 21 Mar 2022 22:28:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2022-03-21-git-intro/</guid><description>&lt;p&gt;From time to time I get to the same place, telling some people about git, what it solves and some basic usage.&lt;/p&gt;
&lt;p&gt;Since I&amp;rsquo;ve done it a lot recenly I wanted to write down a post and enjoy it.&lt;/p&gt;
&lt;h1 id="what-is-git"&gt;What is git?&lt;/h1&gt;
&lt;p&gt;Git is a gift from the gods for the following use cases:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;My laptop is broke! I need the data there is a whole month of work there!&lt;/p&gt;</description></item><item><title>Sbt tests</title><link>https://adrianabreu.com/blog/2022-02-07-sbt-tests/</link><pubDate>Mon, 07 Feb 2022 19:53:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2022-02-07-sbt-tests/</guid><description>&lt;p&gt;Últimamente en el trabajo estoy usando mucho delta para algunas tablas de dimensiones y estas tablas realizan actualizaciones parciales de las filas para replicar la lógica de negocio.&lt;/p&gt;
&lt;p&gt;Esto, nos lleva a varios tests que replican un estado de la tabla y realizan las actualizaciones pertinentes para comprobar todos los flujos y por ende un sobrecoste de ejecución de ese tipo de tests que acaba siendo agotador.&lt;/p&gt;
&lt;p&gt;Una de las soluciones planteadas fue incluir en las builds un parámetro para saltarse el step de ejecución de los tests. Lo cual es legítimo pero al menos para mí, resulta algo arbitrario. Buscando otro concens llegamos a: en las pull request se ejecutarán todos los tests y en el resto de builds (manuales o automáticas de rama) se excluirán estos tests, para que al hacer pruebas o durante las integraciones de las ramas no estemos acumulando tiempo en tests ya validados.&lt;/p&gt;</description></item><item><title>Multiplying rows in Spark</title><link>https://adrianabreu.com/blog/2021-11-11-multiplying-rows-in-spark/</link><pubDate>Thu, 11 Nov 2021 18:32:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2021-11-11-multiplying-rows-in-spark/</guid><description>&lt;p&gt;Earlier this week I checked on a Pull Request that bothered me since I saw it from the first time. Let&amp;rsquo;s say we work for a bank and we are going to give cash to our clients if they get some people to join our bank.&lt;/p&gt;
&lt;p&gt;And we have an advertising campaign definition like this:&lt;/p&gt;
&lt;table&gt;
 &lt;thead&gt;
 &lt;tr&gt;
 &lt;th&gt;campaign_id&lt;/th&gt;
 &lt;th&gt;inviter_cash&lt;/th&gt;
 &lt;th&gt;receiver_cash&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody&gt;
 &lt;tr&gt;
 &lt;td&gt;FakeBank001&lt;/td&gt;
 &lt;td&gt;50&lt;/td&gt;
 &lt;td&gt;30&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;FakeBank002&lt;/td&gt;
 &lt;td&gt;40&lt;/td&gt;
 &lt;td&gt;20&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;FakeBank003&lt;/td&gt;
 &lt;td&gt;30&lt;/td&gt;
 &lt;td&gt;20&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;And then our BI teams defines the schema they want for their dashboards.&lt;/p&gt;</description></item><item><title>The horrible azure devops ui</title><link>https://adrianabreu.com/blog/2021-11-07-the-horrible-azure-devops-pipeline-ui/</link><pubDate>Sun, 07 Nov 2021 16:32:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2021-11-07-the-horrible-azure-devops-pipeline-ui/</guid><description>&lt;p&gt;&lt;strong&gt;Disclaimer: I read the docs, I know this is just complaining and not giving feedback, but man, this UI stills is horrible&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;So&amp;hellip; Let&amp;rsquo;s put into situation, there was a connection update between devops and bitbucket and suddenly most of our pipelines stopped working. They told me to change the connection in the yaml file and that didn&amp;rsquo;t work.&lt;/p&gt;
&lt;p&gt;I know that there are three parts involving in a pipeline for sure:&lt;/p&gt;</description></item><item><title>Regex 101</title><link>https://adrianabreu.com/blog/2021-11-01-regex-101/</link><pubDate>Fri, 05 Nov 2021 16:39:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2021-11-01-regex-101/</guid><description>&lt;p&gt;&lt;em&gt;-You will spent your whole life relearning regex, there is a beginning, but never and end.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Last year I participated in some small code problems and practised some regex. I got used to it and feel quite good at it.&lt;/p&gt;
&lt;p&gt;And today I had to use it again. I had the following dataframe:&lt;/p&gt;
&lt;table&gt;
 &lt;thead&gt;
 &lt;tr&gt;
 &lt;th&gt;product&lt;/th&gt;
 &lt;th&gt;attributes&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody&gt;
 &lt;tr&gt;
 &lt;td&gt;1&lt;/td&gt;
 &lt;td&gt;(SIZE-36)&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;2&lt;/td&gt;
 &lt;td&gt;(COLOR-RED)&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;3&lt;/td&gt;
 &lt;td&gt;(SIZE-38, COLOR-BLUE)&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;4&lt;/td&gt;
 &lt;td&gt;(COLOR-GREEN, SIZE-39)&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;A wonderful set of string merged with properties that could vary. And we wanted one column for each:&lt;/p&gt;</description></item><item><title>Sbt Intro I</title><link>https://adrianabreu.com/blog/2021-10-10-sbt-intro-i/</link><pubDate>Sun, 10 Oct 2021 14:47:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2021-10-10-sbt-intro-i/</guid><description>&lt;p&gt;El mes pasado cambié a otro trabajo :) y por casualidades he vuelto a acabar con proyectos de scala. Este proyecto está bastante avanzado y hace un uso intensivo de los plugins de sbt. De hecho, una tarea que tengo próximamente es hacer una plantilla para proyectos. Así que quería repasar los conceptos básicos de sbt en una serie de posts.&lt;/p&gt;
&lt;p&gt;¿Qué es sbt &lt;strong&gt;scala build tool&lt;/strong&gt;? Es una herramienta para gestionar proyectos en scala. Es la más utilizada (casi el 95% de los proyectos se hacen en sbt) y uno de sus puntos fuertes es que permite trabajar con múltiples versiones de scala haciendo cross-compilation.&lt;/p&gt;</description></item><item><title>A funny bug</title><link>https://adrianabreu.com/blog/2021-07-21-funny-bug/</link><pubDate>Wed, 21 Jul 2021 18:52:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2021-07-21-funny-bug/</guid><description>&lt;p&gt;Ayer mismo estaba intentando analizar unos data_types para completar una exportación al backend. La idea era sencilla, buscar para cada usuario la ultima información disponible en una tabla que representaba actualizaciones sobre su perfil.&lt;/p&gt;
&lt;p&gt;Como a su perfil podías &amp;ldquo;añadir&amp;rdquo;, &amp;ldquo;actualizar&amp;rdquo; y &amp;ldquo;eliminar&amp;rdquo; cosas, pues existian los tres tipos en la tabla. Para que la imaginemos mejor sería tal que así:&lt;/p&gt;
&lt;table&gt;
 &lt;thead&gt;
 &lt;tr&gt;
 &lt;th&gt;user_id&lt;/th&gt;
 &lt;th&gt;favorite_stuff&lt;/th&gt;
 &lt;th&gt;operation&lt;/th&gt;
 &lt;th&gt;metadata&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody&gt;
 &lt;tr&gt;
 &lt;td&gt;A&lt;/td&gt;
 &lt;td&gt;Chocolate&lt;/td&gt;
 &lt;td&gt;Add&lt;/td&gt;
 &lt;td&gt;&amp;hellip;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;A&lt;/td&gt;
 &lt;td&gt;Chocolate&lt;/td&gt;
 &lt;td&gt;Update&lt;/td&gt;
 &lt;td&gt;&amp;hellip;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;B&lt;/td&gt;
 &lt;td&gt;Milk&lt;/td&gt;
 &lt;td&gt;Remove&lt;/td&gt;
 &lt;td&gt;&amp;hellip;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;B&lt;/td&gt;
 &lt;td&gt;Cornflakes&lt;/td&gt;
 &lt;td&gt;Add&lt;/td&gt;
 &lt;td&gt;&amp;hellip;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;De tal manera que habría que combinar todos los eventos para saber cual es el perfil actual del usuario y aplicar cierta lógica. Sin embargo los eventos que habían llegado realmente eran así:&lt;/p&gt;</description></item><item><title>Exportando los datos de firebase</title><link>https://adrianabreu.com/blog/2021-05-06-exporting-firebase-data/</link><pubDate>Thu, 06 May 2021 11:49:36 +0000</pubDate><guid>https://adrianabreu.com/blog/2021-05-06-exporting-firebase-data/</guid><description>&lt;p&gt;Si trabajamos analizando los datos de una aplicación móvil es muy probable que esté integrado algún sistema para trackear los eventos de la app. Y entre ellos, uno de los más conocidos es Firebase.&lt;/p&gt;
&lt;p&gt;Estos eventos contienen mucha información útil y nos permiten por ejemplo saber, un usuario que se ha ido cuanto tiempo ha usado la aplicación o cuantos dias han pasado.&lt;/p&gt;
&lt;p&gt;O si realmente ha seguido el flujo de acciones que esperabamos (con un diagrama de sankey podríamos ver donde se han ido los usuarios).&lt;/p&gt;</description></item><item><title>Notas sobre storytelling with data</title><link>https://adrianabreu.com/blog/2021-01-04-notes-storytelling-with-data-/</link><pubDate>Mon, 04 Jan 2021 19:00:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2021-01-04-notes-storytelling-with-data-/</guid><description>&lt;p&gt;Como uno de los objetivos antes de cambiar de año quería empezar a dar visibilidad sobre el producto en el que estoy trabajando con un dashboard. Tras probar varias opciones, hemos optado por utilizar &lt;a href="https://aws.amazon.com/es/quicksight/"&gt;Quicksight&lt;/a&gt; para simplificar los procesos en aws y reducir nuestra infraestructura.&lt;/p&gt;
&lt;p&gt;Aún así, empezando un dashboard de cero, es muy difícil transmitir la información de forma clara. Es importante evitar que los usuarios vengan simplemente a expotar sus datos a csv para luego cargarlos en excel.&lt;/p&gt;</description></item><item><title>Configurando poetry y gitlab</title><link>https://adrianabreu.com/blog/2020-12-30-configurando-poetry-gitlab/</link><pubDate>Wed, 30 Dec 2020 15:42:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2020-12-30-configurando-poetry-gitlab/</guid><description>&lt;p&gt;Hace poco más de un mes cambié de trabajo y me encontré además con un cambio de stack considerable. Ahora estoy trabajando con aws + github + python. Y bueno, al margen de los cambios de conceptos y demás, me ha llevado bastante encontrar un flujo de trabajo que no me pareciera &amp;ldquo;frágil&amp;rdquo;.&lt;/p&gt;
&lt;p&gt;Lo primero y que me ha decepcionado bastante es que github no incluye soporte para hostear paquetes de python. Lo tendrá sí, pero sin fecha clara y &lt;a href="https://github.com/github/roadmap/projects/1?card_filter_query=label%3Apackages"&gt;por lo pronto parece que será despues de Junio de 2021&lt;/a&gt;.&lt;/p&gt;</description></item><item><title>Jugando con Data Factory</title><link>https://adrianabreu.com/blog/2020-10-01-jugando-con-df/</link><pubDate>Thu, 01 Oct 2020 10:12:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2020-10-01-jugando-con-df/</guid><description>&lt;p&gt;Sorprendentemente, hasta ahora, no había tenido la posibilidad de trabajar con data factory, sólo lo habia usado para algunas migraciones de datos.&lt;/p&gt;
&lt;p&gt;Sin embargo, tras estabilizar un proyecto y consolidar su nueva etapa, necesitabamos simplificar la solución implementada para migrar datos.&lt;/p&gt;
&lt;p&gt;Una representación sencilla de la arquitectura actual sería:&lt;/p&gt;
&lt;p&gt;&lt;img src="https://adrianabreu.com/images/data-factory/original-architecture.png" alt="Arquitectura actual"&gt;&lt;/p&gt;
&lt;p&gt;En un flujo muy sencillo sería esto:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;La etl escribe un fichero csv con spark en un directorio de un blob storage.&lt;/li&gt;
&lt;li&gt;La primera function filtra los ficheros de spark que no son part- y se encarga de notificar a una function que actua de gateway para el batch con que fichero queremos enviar, el nombre original, el path y el nombre que queremos darle.&lt;/li&gt;
&lt;li&gt;Esta function de gateway se encarga de realizar las llamadas necesarias a la api de Azure para generar una tarea en el batch.&lt;/li&gt;
&lt;li&gt;El batch se encarga de comprimir el fichero y enviarlo al sftp del cliente, recuperando las credenciales según el tipo de fichero que se trate.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Este proceso nos permitía trabajar con dos versiones del proyecto en lo que hacíamos la migración a la nueva versión. Ahora que la nueva versión ya está consolidada y hemos conseguido además que el cliente utilice un formato de compresión que podemos escribir directamente desde spark sin recurrir al batch, es el momento de cambiar la arquitectura de transferencia de datos.&lt;/p&gt;</description></item><item><title>Tipos de join en spark</title><link>https://adrianabreu.com/blog/2020-12-29-spark-joins/</link><pubDate>Tue, 29 Sep 2020 18:52:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2020-12-29-spark-joins/</guid><description>&lt;p&gt;Hace unos días tuve la fortuna (o desgracia) de implementar la lógica más compleja de todo el dominio.
El resultado, como esperaba, una etl que falaba por recursos constantementes. El problema:&lt;/p&gt;
&lt;pre tabindex="0"&gt;&lt;code&gt;Caused by: org.apache.spark.SparkException: Could not execute broadcast in 300 secs. You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or disable broadcast join by setting spark.sql.autoBroadcastJoinThreshold to -1
&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Lo primero fue revisar el plan de ejecución para ver que estaba sucediendo.&lt;/p&gt;</description></item><item><title>Calcular el domingo de la semana</title><link>https://adrianabreu.com/blog/2020-09-02-calcular-el-domingo-correspondiente/</link><pubDate>Wed, 02 Sep 2020 10:12:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2020-09-02-calcular-el-domingo-correspondiente/</guid><description>&lt;p&gt;A la hora de publicar reportes es común agrupar los datos por semanas. Otro motivo es alinearse con el negocio donde los cierres pueden producirse en días concretos, por ejemplo, un domingo.&lt;/p&gt;
&lt;p&gt;En esos casos si tenemos los datos particionados por días nos interesa saber a que domingo correspondería cada uno de los datos.&lt;/p&gt;
&lt;p&gt;Los que venimos de otros entornos tendemos a pensar en esas complicadas librerías de fechas (moment.js, jodatime, etc). Incluso alguien podría pensar en extraer los datos del dataframe y procesarlo en local.&lt;/p&gt;</description></item><item><title>Detectando ficheros pequenos Spark</title><link>https://adrianabreu.com/blog/2020-08-25-detectando-ficheros-pequenos/</link><pubDate>Tue, 25 Aug 2020 17:22:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2020-08-25-detectando-ficheros-pequenos/</guid><description>&lt;p&gt;Uno de los mayores problemas de rendimiento que podemos encontrar en los datalake es tener que mover una enorme cantidad de ficheros pequeños, por el overhead que eso representa en las transacciones.
Este post de databricks recomendada &lt;a href="https://forums.databricks.com/questions/101/what-is-an-optimal-size-for-file-partitions-using.html"&gt;https://forums.databricks.com/questions/101/what-is-an-optimal-size-for-file-partitions-using.html&lt;/a&gt; que se crearan ficheros de 1GB parquet.&lt;/p&gt;
&lt;p&gt;Sin embargo mucha gente no sabe como detectar esto. Hace poco estuve jugando con un notebook y usando simplemente las herramientas del dbutils pude clasificar los ficheros que tenia en las entidades del datalake en múltiples categorías, así podría estimar cuantos ficheros había en un rango de tiempo.&lt;/p&gt;</description></item><item><title>Spark windows functions (I)</title><link>https://adrianabreu.com/blog/2020-08-11-spark-windows-functions/</link><pubDate>Tue, 11 Aug 2020 18:52:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2020-08-11-spark-windows-functions/</guid><description>&lt;p&gt;En analítica, es muy común hacer uso de las funciones de ventana para distintos cálculos. Hace poco me encontré con un pequeño problema cuya solución mejoró muchísimo al usar las funciones de ventana, demos un poco de contexto.&lt;/p&gt;
&lt;p&gt;Tenemos una dimensión de usuarios donde los usuarios se van registrando con una fecha y tenemos una tabla de ventas donde tenemos las ventas globales para cada día&lt;/p&gt;
&lt;p&gt;Y lo que queremos dar es una visión de cómo cada día evoluciona el programa, para ello se quiere que cada día estén tanto las ventas acumuladas como los registros acumulados.&lt;/p&gt;</description></item><item><title>Acceso al keyvault mediante certificados</title><link>https://adrianabreu.com/blog/2020-05-31-acceso-keyvault-certificados/</link><pubDate>Sat, 30 May 2020 19:52:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2020-05-31-acceso-keyvault-certificados/</guid><description>&lt;p&gt;En el proceso de migración de una aplicación de webjob a azure batch, nos encontramos con la problemática de gestionar los secretos. El servicio de batch se encarga de recoger una aplicación de un storage y hacer ejecuciones de ellas (tasks) en unas máquinas concretas (pool).&lt;/p&gt;
&lt;p&gt;Para poder gestionar los secretos de la aplicación, estos estaban guardados en keyvault. Y teníamos que acceder de forma segura a ello. Por eso optamos por utilizar la autenticación via certificado. La idea de este tutorial es reproducir los mismos pasos que he usado yo para poder usar este certificado.&lt;/p&gt;</description></item><item><title>Scala best practices notes</title><link>https://adrianabreu.com/blog/2020-04-27-scala-best-practices-nicolas-rinaudo/</link><pubDate>Mon, 27 Apr 2020 20:55:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2020-04-27-scala-best-practices-nicolas-rinaudo/</guid><description>&lt;p&gt;He aprovechado estos días de cuarentena para revisar algunos de los &amp;ldquo;huecos&amp;rdquo; de conocimiento que tenía en Scala. Una de las charlas que he podido ver es esta: &lt;a href="https://www.youtube.com/watch?v=DGa58FfiMqc"&gt;Scala best practices I wish someone&amp;rsquo;d told me about - Nicolas Rinaudo&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Por supuesto siempre recomiendo ver la charla, pero he querido condensar (aún más) ese conocimiento en este post, insisto, es amena y muy interesante, muchos de los puntos que se definen en la charla no se han explicado porque la mayoría se resuelven en dotty y aunque&lt;/p&gt;</description></item><item><title>Notas sobre programación funcional en Scala I</title><link>https://adrianabreu.com/blog/2020-04-07-notes-on-functional-programming-in-scala-i/</link><pubDate>Mon, 06 Apr 2020 18:22:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2020-04-07-notes-on-functional-programming-in-scala-i/</guid><description>&lt;p&gt;Hace unos días pude comprarme el &lt;a href="https://www.amazon.es/Functional-Programming-Scala-Paul-Chiusano/dp/1617290653"&gt;libro de Paul Chiusano y Rúnar Bjarnason: Functional Programming in scala&lt;/a&gt; y no puedo estar más contento con él.&lt;/p&gt;
&lt;p&gt;Como ya es costumbre, aprovecho para dejar mis notas sobre el libro en el blog. No se trata de un resumen del mismo sino curiosidades que sé que volveré a consultar en un futuro. Intentaré que no queden post excesivamente largos haciendo un por capítulo. Igualmente, recomiendo a todo el mundo adquirir &amp;ldquo;el libro rojo de Scala&amp;rdquo; y echarle un vistazo.&lt;/p&gt;</description></item><item><title>Límites en azure functions para procesos de larga duración</title><link>https://adrianabreu.com/blog/2020-02-24-azure-functions-service-bus-limits/</link><pubDate>Mon, 24 Feb 2020 19:22:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2020-02-24-azure-functions-service-bus-limits/</guid><description>&lt;p&gt;Estas últimas semanas he tenido que implementar ciertas mejoras en un proyecto. El objetivo era muy simple, conectar el proyecto a una aplicación de datawarehousing existente, y de forma externa, realizar agregados y luego aplicar cierto procesamiento para un servicio en particular.&lt;/p&gt;
&lt;p&gt;Además había una serie de requisitos extras:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;El procesamiento iba a ser reutilizado por otro proyecto. Y requería comprimir y cifrar archivos grandes.&lt;/li&gt;
&lt;li&gt;La primera parte tenía que simplemente,&lt;/li&gt;
&lt;li&gt;Había una deadline muy cercana para este proyecto.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Con todas estas limitaciones, la solución propuesta fue esta:&lt;/p&gt;</description></item><item><title>Conceptos básicos de Spark</title><link>https://adrianabreu.com/blog/2019-11-09-spark-concepts-basicos/</link><pubDate>Sat, 09 Nov 2019 19:22:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2019-11-09-spark-concepts-basicos/</guid><description>&lt;p&gt;&lt;strong&gt;Nota del autor&lt;/strong&gt;: Todos los contenidos de este artículo son extractos del libro &amp;ldquo;The Data Engineer&amp;rsquo;s Guide to Apache Spark&amp;rdquo; que puedes descargar desde la pagina de databricks: &lt;a href="https://databricks.com/lp/ebook/data-engineer-spark-guide"&gt;https://databricks.com/lp/ebook/data-engineer-spark-guide&lt;/a&gt;&lt;/p&gt;
&lt;h2 id="preludio"&gt;Preludio:&lt;/h2&gt;
&lt;h3 id="cluster"&gt;Cluster:&lt;/h3&gt;
&lt;p&gt;Un cluster no es más que un conjunto de máquinas trabajando de forma coordinada. Un cluster de Spark se compone de nodos. Uno actúa como DRIVER y es el punto de entrada para el código del usuario. Los otros actúan como EXECUTOR que seran los encargados de realizar las operaciones.&lt;/p&gt;</description></item><item><title>Empezando en Spark con Docker</title><link>https://adrianabreu.com/blog/2019-11-10-empezando-en-spark-con-docker/</link><pubDate>Sat, 09 Nov 2019 19:22:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2019-11-10-empezando-en-spark-con-docker/</guid><description>&lt;p&gt;A pesar de haber leído guías tan buenas como:&lt;/p&gt;
&lt;p&gt;&lt;a href="https://medium.com/@bogdan.cojocar/how-to-run-scala-and-spark-in-the-jupyter-notebook-328a80090b3b"&gt;https://medium.com/@bogdan.cojocar/how-to-run-scala-and-spark-in-the-jupyter-notebook-328a80090b3b&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href="https://medium.com/@singhpraveen2010/install-apache-spark-and-configure-with-jupyter-notebook-in-10-minutes-ae120ebca597"&gt;https://medium.com/@singhpraveen2010/install-apache-spark-and-configure-with-jupyter-notebook-in-10-minutes-ae120ebca597&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Se me ha hecho cuesta arriba el poder conectar un notebook de jupyter y utilizar Scala. Entre configurar el apache toree para poder usar scala en los notebooks y algún error luego en spark al usarlo desde IntelliJ, me he acabado rindiendo.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Nota del autor:&lt;/em&gt; Como disclaimer esto ocurre probablemente porque estoy en Manjaro y mi version de Scala es incompatible. Esta clase de problemas en su día las solucionaba fijando una versión, sin embargo, creo que teniendo una herramienta tan potente como es Docker estoy reinventando la rueda par un problema ya resuelto. Además de que voy a probarlo también en un windows para ver que es una solución agnóstical SO.&lt;/p&gt;</description></item><item><title>Correlated subqueries</title><link>https://adrianabreu.com/blog/2019-09-26-correlated-subqueries/</link><pubDate>Thu, 26 Sep 2019 20:43:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2019-09-26-correlated-subqueries/</guid><description>&lt;p&gt;Llevo un par de meses viendo como la mayoría de esfuerzos en el proyecto en el que estoy se centran en evitar los joins en las distintas capas de análisis. Aprovechando las capacidades de spark se busca tener las estructuras muy desnormalizadas y se había &amp;ldquo;endemoniado&amp;rdquo; al join considerarlo perjudicial.&lt;/p&gt;
&lt;p&gt;Tanto es así que llevo un par de días peleando con una pieza de código que me ha sorprendido. Partiendo de una tabla de hechos que agrupa datos para un periodo a hasta b, se quiere que se &amp;ldquo;colapsen&amp;rdquo; los datos de hace 14 días. Será mejor con un ejemplo:&lt;/p&gt;</description></item><item><title>Datos I - Introducción al Datawarehousing</title><link>https://adrianabreu.com/blog/2019-02-05-data-i-introduccion-al-datawarehouse/</link><pubDate>Tue, 05 Feb 2019 14:43:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2019-02-05-data-i-introduccion-al-datawarehouse/</guid><description>&lt;p&gt;En los últimos meses mi trabajo ha pivotado del mundo de la web al mundo de los datos. He entrado a participar en un proyecto de Data Warehouse y he acabado muy contento en él. Hace unos días mi cambio se oficializó completamente y ahora me he dado cuenta de que no solo tengo un mundo técnico ante mí, sino que además necesito consolidar algunas bases teóricas.&lt;/p&gt;
&lt;p&gt;Investigando la bibliografía, me han recomendado en Reddit: &lt;em&gt;The Data Warehouse Toolkit, The Complete Guide to Dimensional Modeling 2nd Edition.&lt;/em&gt;
Y el libro parece encajar perfectamente en el conocimiento que busco. Aun así, como todo, por necesidad, intentaré resumir en unas cuantas entradas el conocimiento que se puede obtener de este libro. El cual recomiendo encarecidamente.&lt;/p&gt;</description></item><item><title>Angular Series III - Dynamic components</title><link>https://adrianabreu.com/blog/2017-12-17-angular-series-iii-dynamic-components/</link><pubDate>Sun, 17 Dec 2017 14:43:32 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-12-17-angular-series-iii-dynamic-components/</guid><description>&lt;p&gt;Antes de terminar repasando el tema de &lt;em&gt;templating&lt;/em&gt;, quiero hacer un inciso. Existen ciertos casos donde el templating es insuficiente y lo que necesitamos es simplemente escoger dinámicamente que componente vamos a renderizar.&lt;/p&gt;
&lt;p&gt;Esto está documentado en la documentación de angular bajo el nombre de &lt;a href="https://angular.io/guide/dynamic-component-loader"&gt;Dynamic Components&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id="cómo-funcionan-estos-dynamics-components"&gt;¿Cómo funcionan estos dynamics components?&lt;/h3&gt;
&lt;p&gt;Explicado mal y pronto, la idea es: Escoger un elemento de la vista que actue de contenedor e inyectar el componente debe ir ahí.&lt;/p&gt;</description></item><item><title>Angular Series II - Templating</title><link>https://adrianabreu.com/blog/2017-11-18-angular-series-ii-templating/</link><pubDate>Sat, 18 Nov 2017 18:53:17 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-11-18-angular-series-ii-templating/</guid><description>&lt;p&gt;Continuando con el &lt;a href="https://adrianabreu.com/blog/2017-08-14-angular-series-i-transclusion/"&gt;artículo del otro día sobre proyección de contenido&lt;/a&gt; aquí pretendo mostrar otra forma de pasar contenido: las templates.
¿Qué es una template? Es un pedazo de html envuelto entre entre etiquetas ng-template tal que así:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-html" data-lang="html"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&amp;lt;&lt;span style="color:#f92672"&gt;ng-template&lt;/span&gt;&amp;gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&amp;lt;&lt;span style="color:#f92672"&gt;div&lt;/span&gt; &lt;span style="color:#a6e22e"&gt;class&lt;/span&gt;&lt;span style="color:#f92672"&gt;=&lt;/span&gt;&lt;span style="color:#e6db74"&gt;”as-template”&lt;/span&gt;&amp;gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt; I won’t be rendered
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&amp;lt;/&lt;span style="color:#f92672"&gt;div&lt;/span&gt;&amp;gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;&amp;lt;/&lt;span style="color:#f92672"&gt;ng-template&lt;/span&gt;&amp;gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Si esto lo ponemos en un componente, tal como en el siguiente ejemplo en el navegador aparecerá: NADA.&lt;/p&gt;
&lt;iframe class="aa_iframes" src="https://stackblitz.com/edit/angular-template-series?ctl=1&amp;embed=1&amp;file=app/parent/parent1.component.ts&amp;view=preview"&gt;&lt;/iframe&gt;
&lt;p&gt;&lt;strong&gt;¿Por qué?&lt;/strong&gt; Porque las plantillas de angular no se renderizan al ser evaluadas. Se renderizan donde una directiva les indique.&lt;/p&gt;</description></item><item><title>Experimentando con Redux</title><link>https://adrianabreu.com/blog/2017-11-06-experimentando-con-redux/</link><pubDate>Mon, 06 Nov 2017 23:18:17 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-11-06-experimentando-con-redux/</guid><description>&lt;p&gt;Con el objetivo de mejorar el &lt;a href="https://github.com/etsiiull/SIMDE"&gt;simulador SIMDE&lt;/a&gt; me decidí a probar un sistema de gestión de estados y concentrar el funcionamiento de la aplicación. ¿El motivo? Era la única forma sensata que tenía de poder gestionar la UI cuando entrara la máquina VLIW sin que todo fuera un caos.&lt;/p&gt;
&lt;p&gt;Para ello he recurrido a mi aplicación favorita: Chuck Norris client app. (Ya la he hecho en AngularJS y Angular previamente). Como boilerplate he utilizado el template de visual studio, ya que quería tener compatibilidad con typescript.&lt;/p&gt;</description></item><item><title>Un breve sumario</title><link>https://adrianabreu.com/blog/2017-10-08-breve-sumario/</link><pubDate>Sun, 08 Oct 2017 18:28:17 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-10-08-breve-sumario/</guid><description>&lt;p&gt;El mes de septiembre ha sido un poco caótico, pero ya creo que me he adaptado a la rutina. Por lo pronto he empezado con el blog, ahora el auto despliegue va mucho mejor, estoy utilizando CircleCI en vez de Wecker.&lt;/p&gt;
&lt;p&gt;Por otra parte, el diseño del blog se ha separado por mucho del tema original, y no contento con esto he mejorado el desarrollo del tema permitiendo el uso de Sass.&lt;/p&gt;</description></item><item><title>SOLID Principles</title><link>https://adrianabreu.com/knowledge/solidprinciples/</link><pubDate>Sun, 08 Oct 2017 18:04:42 +0000</pubDate><guid>https://adrianabreu.com/knowledge/solidprinciples/</guid><description>&lt;h2 id="srp-single-responsability-principle"&gt;SRP: Single Responsability Principle&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Una pieza de software debería tener una única razón para cambiar.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Si una clase tiene más de una &amp;ldquo;responsabilidad&amp;rdquo; (razón de cambio), un cambio en algún requisito podría ser muy difícil de modelar. Ya que al cumplir con esta responsabilidad podríamos estar incumpliendo otras.&lt;/p&gt;
&lt;p&gt;Esto hace que el diseño sea realmente frágil y esté acoplado: es decir, se va a romper de formas inesperadas.&lt;/p&gt;
&lt;p&gt;Es importante ver que esta regla aunque es general no implica que tengamos que desgranar siempre todas las clases. Por ejemplo si una de mis clases se gestiona en base a una lógica establecida en la constitución española, no creo que esa lógica vaya a cambiar, con lo cual no es una razón de cambio y no pasa nada porque esté ahí.&lt;/p&gt;</description></item><item><title>Angular series I - Proyección de contenido (Content projection)</title><link>https://adrianabreu.com/blog/2017-08-14-angular-series-i-transclusion/</link><pubDate>Mon, 14 Aug 2017 17:32:37 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-08-14-angular-series-i-transclusion/</guid><description>&lt;p&gt;Cuando creé este blog tenía bastante claro que uno de los objetivos principales era que me sirviera de utilidad para interiorizar lo que voy aprendiendo.
Y aunque he escrito ciertas cosas útiles para mi día a día no estoy registrando ni una centésima parte de la información que mi mente ha ido procesando estos
meses.&lt;/p&gt;
&lt;p&gt;Así que voy a dedicarme a escribir un artículo semanal sobre un tema con el que llevo ya casi un año: &lt;strong&gt;Angular&lt;/strong&gt;.&lt;/p&gt;</description></item><item><title>Auto deployment en gh-pages con Travis</title><link>https://adrianabreu.com/blog/2017-08-04-autodeploy-con-travis/</link><pubDate>Fri, 04 Aug 2017 16:37:11 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-08-04-autodeploy-con-travis/</guid><description>&lt;p&gt;Una de las mejores características de github sin duda alguna, son las &lt;a href="https://pages.github.com/"&gt;gh-pages&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Las gh-pages nos permiten desplegar el código de nuestra aplicación frontend a través de esta rama, de tal forma que muchos de nuestros proyectos (por ejemplo este blog) estén disponibles sin tener limitaciones de hosting.&lt;/p&gt;
&lt;p&gt;Pero sin duda una desventaja es el hecho de tener que mantener el deploy de nuestras revisiones: cambiar de rama, eliminar el contenido, hacer una build y desplegar.&lt;/p&gt;</description></item><item><title>Mi experiencia con React</title><link>https://adrianabreu.com/blog/2017-07-28-mi-experiencia-con-react/</link><pubDate>Fri, 28 Jul 2017 17:11:10 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-07-28-mi-experiencia-con-react/</guid><description>&lt;p&gt;Hoy se cumple un mes desde que entregué mi trabajo de fin de grado. &lt;em&gt;(Modestia aparte, conseguí un 10)&lt;/em&gt;. Han pasado muchas cosas desde entonces, como que por ejemplo ahora mismo estoy viviendo en Barcelona y que trabajo para nada más y nada menos que &lt;strong&gt;Plain Concepts&lt;/strong&gt;. Pero al margen de eso, vamos a centrarnos en mi trabajo de fin de grado.&lt;/p&gt;
&lt;p&gt;En tercero de grado de ingeniería informática es posible escoger una especialidad. Yo descontento con la mayoría me decidí por Ingeniería de Computadores &lt;em&gt;a.k.a.&lt;/em&gt; Hardware. Y a lo largo de estos dos cursos me encontré con dos asignaturas de &lt;strong&gt;Arquitectura de Computadores&lt;/strong&gt;.&lt;/p&gt;</description></item><item><title>Empezar a programar</title><link>https://adrianabreu.com/blog/2017-06-04-empezar-a-programar/</link><pubDate>Sun, 04 Jun 2017 13:07:30 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-06-04-empezar-a-programar/</guid><description>&lt;p&gt;Creo que me he visto en más de una decena de ocasiones aconsejando a gente como empezar en el mundo
de la programación, así que esta entrada viene dedicada a todos ellos.&lt;/p&gt;
&lt;p&gt;Por supuesto se trata de una guía basada en mi opinión y solo mi opinión.&lt;/p&gt;
&lt;h3 id="daw-o-dam"&gt;¿DAW o DAM?&lt;/h3&gt;
&lt;p&gt;Realmente mi respuesta es bastante clara: &lt;strong&gt;DAW&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;El mundo web cada vez va tomando más y más fuerza.
Ya se pueden desarrollar aplicaciones de escritorio utilizando como base aplicaciones web con tecnologías
como &lt;a href="https://electron.atom.io/"&gt;electron&lt;/a&gt; (un buen ejemplo de este tipo de aplicaciones &lt;a href="https://code.visualstudio.com/"&gt;Visual Studio Code&lt;/a&gt;). Y además también
se pueden desarrollar aplicaciones móviles con &lt;a href="https://facebook.github.io/react-native/"&gt;React Native&lt;/a&gt; e &lt;a href="https://ionicframework.com/"&gt;Ionic&lt;/a&gt;.&lt;/p&gt;</description></item><item><title>Docker</title><link>https://adrianabreu.com/blog/2017-04-09-docker/</link><pubDate>Sun, 09 Apr 2017 15:20:44 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-04-09-docker/</guid><description>&lt;h2 id="qué-es-docker"&gt;¿Qué es docker?&lt;/h2&gt;
&lt;p&gt;Docker es un software que permite automatizar el despliegue de aplicaciones utilizando contenedores.&lt;/p&gt;
&lt;p&gt;Los contenedores hacen uso de la virtualización a nivel de sistema operativo, con lo cual son mucho más ligeros y rápidos que las máquinas virtuales.&lt;/p&gt;
&lt;h2 id="para-qué-lo-puedo-necesitar"&gt;¿Para qué lo puedo necesitar?&lt;/h2&gt;
&lt;p&gt;En el día a día, una de las cosas para las que Docker me resulta más útil sin duda es para desplegar las bases de datos en desarrollo.&lt;/p&gt;</description></item><item><title>Capas en el backend</title><link>https://adrianabreu.com/blog/2017-04-08-capas-en-el-backend/</link><pubDate>Sat, 08 Apr 2017 17:33:10 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-04-08-capas-en-el-backend/</guid><description>&lt;p&gt;Una de las cosas más difíciles cuando ves un proyecto de backend por primera vez es discernir la funcionalidad y responsabilidad de las distintas capas. Así que me he planteado dar una visión general.&lt;/p&gt;
&lt;p&gt;Antes de nada, quiero aclarar que es probable que esta nomenclatura no coincida con otra que veais por ahí, existen muchos sinónimos para los mismos conceptos, yo por mi parte, voy a explicar cual utilizo actualmente.&lt;/p&gt;</description></item><item><title>Interacción a través de ViewChild en Angular</title><link>https://adrianabreu.com/blog/2017-03-24-interaccion-a-traves-de-viewchild/</link><pubDate>Fri, 24 Mar 2017 23:11:22 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-03-24-interaccion-a-traves-de-viewchild/</guid><description>&lt;p&gt;Hoy me he encontrado con una situación peculiar en un código. Aunque considero que quizás como está realizada la tarea no sea la mejor opción, creo que es un buen ejemplo para entender algunos conceptos de Angular.&lt;/p&gt;
&lt;p&gt;Partiendo de una aplicaicón muy básica donde tenemos dos componentes: &lt;strong&gt;AppComponent&lt;/strong&gt; y &lt;strong&gt;ChildComponent&lt;/strong&gt;, vamos a renderizar dinámicamente el componente hijo desde el componente padre y ejecutar una serie de acciones.&lt;/p&gt;
&lt;p&gt;Empecemos por el componente padre:&lt;/p&gt;</description></item><item><title>Como construir un portfolio</title><link>https://adrianabreu.com/blog/2017-03-12-como-construir-un-portfolio/</link><pubDate>Sun, 12 Mar 2017 10:39:22 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-03-12-como-construir-un-portfolio/</guid><description>&lt;p&gt;Desde hace un tiempo en los países anglosajones los desarrolladores tienen una herramienta más importante que su CV, el portfolio.&lt;/p&gt;
&lt;p&gt;Un portfolio no es más que una muestra de tus trabajos y una justificación de las habilidades escritas en tu CV. A día de hoy, es increíblemente fácil tener un portfolio. Pero ya que voy a hablar de eso, aprovecharé para hablar también de qué debería contener un portfolio, y como enfocar el portfolio de un desarrollador junior proporcionando algunas ideas básicas.&lt;/p&gt;</description></item><item><title>Sin componentes a componentes</title><link>https://adrianabreu.com/blog/2017-03-11-sin-componentes-a-componentes/</link><pubDate>Sat, 11 Mar 2017 17:44:11 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-03-11-sin-componentes-a-componentes/</guid><description>&lt;p&gt;Hace unos meses mientras me formaba en Angular 1 hice con un amigo un cliente para una api que proporcionaba frases de Chuck Norris.&lt;/p&gt;
&lt;p&gt;Fue un desarrollo divertido donde almacenábamos las frases descargadas en localStorage y permítiamos filtrar las frases por categorías.&lt;/p&gt;
&lt;p&gt;El diseño era modular y me quedé contento con lo que aprendí. Pero entonces entré en prácticas en la empresa y me puse a formarme en Angular 2.&lt;/p&gt;</description></item><item><title>Quizás deberías empezar un blog</title><link>https://adrianabreu.com/blog/2017-03-04-quizas-deberias-empezara-un-blog/</link><pubDate>Sat, 04 Mar 2017 19:46:22 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-03-04-quizas-deberias-empezara-un-blog/</guid><description>&lt;p&gt;Me encanta leer artículos de programación y de tecnología. Leo muchísimos artículos.&lt;/p&gt;
&lt;p&gt;Es más, considero que paso al menos de media, 2 horas diarias leyendo esta clase de artículos.&lt;/p&gt;
&lt;p&gt;Pero soy otro lector invisible más, no devuelvo lo que aprendo a la comunidad (aunque ahora me he ido animando con los comentarios) y lo peor, no lo almaceno.&lt;/p&gt;
&lt;p&gt;Un blog puede ser una parte más de tu portfolio, puede recoger tus dudas y tu avance. Recuerdo, leer muchos artículos del blog de &lt;a href="https://giltesa.com/"&gt;giltesa&lt;/a&gt;, aunque en aquellos momentos yo estaba programando en pañales, me gustaba mucho ver los pequeños códigos en C que ponía.&lt;/p&gt;</description></item><item><title>Probando Hugo</title><link>https://adrianabreu.com/blog/2017-02-05-probando-hugo/</link><pubDate>Sun, 05 Feb 2017 17:31:42 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-02-05-probando-hugo/</guid><description>&lt;p&gt;La verdad es que estoy asombrado. Hexo me tenía muy contento, pero aún así el rendimiento no me convencía tanto como esperaba. Y me pasaba los días dando largas hasta ponerme a postear. Aprovechando que quería hacer una pequeña limpieza antes de ponerme con el trabajo de fin de grado, he estado mirando este generador, Hugo.&lt;/p&gt;
&lt;p&gt;Este generador basado en el lenguaje Go, es increíblemente potente, portable, y sencillo de instalar. Sinceramente, tras probar Jekyll, Hexo y Hugo, sin duda me quedo con este último. Habia visto otros como Octopress, pero tampoco llegó a convencerme.&lt;/p&gt;</description></item><item><title>Contact me</title><link>https://adrianabreu.com/about/</link><pubDate>Sun, 05 Feb 2017 16:28:10 +0000</pubDate><guid>https://adrianabreu.com/about/</guid><description>&lt;p&gt;If you want to reach me you can do through:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/adrianabreu"&gt;Github&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;a href="https://twitter.com/adrianabreudev"&gt;Twitter&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;a href="https://linkedin.com/in/adrianabreu"&gt;Linkedin&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;O incluso a través de mi correo electrónico: &lt;strong&gt;&lt;a href="mailto:adrianabreugonzalez@outlook.com"&gt;adrianabreugonzalez@outlook.com&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;</description></item><item><title>Nuevos comienzos</title><link>https://adrianabreu.com/blog/2017-01-08-nuevos-comienzos/</link><pubDate>Sun, 08 Jan 2017 10:22:36 +0000</pubDate><guid>https://adrianabreu.com/blog/2017-01-08-nuevos-comienzos/</guid><description>&lt;p&gt;No creía que fuera a estar escribiendo esto tan pronto, pero hace poco terminé de realizar las prácticas externas y ya tengo un trabajo en esa misma empresa.&lt;/p&gt;
&lt;p&gt;Mis labores, al igual que en las prácticas, consiste en desarrollar utilizando Angular 2 y Java.&lt;/p&gt;
&lt;p&gt;He ido aprendiendo (y sigo, y seguiré) muchísimo. Y es por eso que necesito este blog. Un lugar donde compartir toda la información que voy absorbiendo sobre esta tecnología que cada día me gusta más.&lt;/p&gt;</description></item></channel></rss>