Skip to content

Commit c75f573

Browse files
committed
ensure partition by country
1 parent 6d815aa commit c75f573

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

quotaclimat/data_processing/mediatree/s3/api_to_s3.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,14 +193,15 @@ def save_to_s3(
193193
based_path = "s3/parquet"
194194

195195
try:
196-
if country==FRANCE: # You need this for france because of legacy partitioning either way
196+
if country==FRANCE and set_filename: # You need this for france because of legacy partitioning either way
197197
df['country'] = country.name
198198
# add partition columns year, month, day to dataframe
199199
if not set_filename:
200200
df['year'] = date.year
201201
df['month'] = date.month
202202
df['day'] = date.day
203203
df['channel'] = channel
204+
df['country'] = country.name
204205

205206
df = df._to_pandas() # collect data accross ray workers to avoid multiple subfolders
206207

0 commit comments

Comments
 (0)