Interrupção no processamento dia 05/05 por ausência de data em fascículo

[robertatakenaka]

Descrição do problema

https://airflow.scielo.br/admin/airflow/log?task_id=process_issues_task&dag_id=sync_isis_to_kernel&execution_date=2023-05-05T19%3A10%3A37%2B00%3A00&format=json

*** Reading local file: /usr/local/airflow/logs/sync_isis_to_kernel/process_issues_task/2023-05-05T19:10:37+00:00/6.log
[2023-05-05 19:40:02,827] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: sync_isis_to_kernel.process_issues_task 2023-05-05T19:10:37+00:00 [queued]>
[2023-05-05 19:40:02,904] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: sync_isis_to_kernel.process_issues_task 2023-05-05T19:10:37+00:00 [queued]>
[2023-05-05 19:40:02,904] {taskinstance.py:880} INFO - 
--------------------------------------------------------------------------------
[2023-05-05 19:40:02,905] {taskinstance.py:881} INFO - Starting attempt 6 of 6
[2023-05-05 19:40:02,905] {taskinstance.py:882} INFO - 
--------------------------------------------------------------------------------
[2023-05-05 19:40:02,945] {taskinstance.py:901} INFO - Executing <Task(PythonOperator): process_issues_task> on 2023-05-05T19:10:37+00:00
[2023-05-05 19:40:02,953] {standard_task_runner.py:54} INFO - Started process 117389 to run task
[2023-05-05 19:40:03,074] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'sync_isis_to_kernel', 'process_issues_task', '2023-05-05T19:10:37+00:00', '--job_id', '237378', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/sync_isis_to_kernel.py', '--cfg_path', '/tmp/tmp7o5c6vw0']
[2023-05-05 19:40:03,078] {standard_task_runner.py:78} INFO - Job 237378: Subtask process_issues_task
[2023-05-05 19:40:03,229] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: sync_isis_to_kernel.process_issues_task 2023-05-05T19:10:37+00:00 [running]> airflow-8696c4d7b7-jxdgr
[2023-05-05 19:40:03,635] {sync_isis_to_kernel.py:372} INFO - reading file from /usr/local/airflow/work/manual__2023-05-05T19:10:37+00:00/json/issue.json.
[2023-05-05 19:40:16,148] {taskinstance.py:1150} ERROR - strptime() argument 1 must be str, not None
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 113, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/airflow/dags/sync_isis_to_kernel.py", line 377, in process_issues
    issues_as_kernel = [issue_as_kernel(issue) for issue in issues]
  File "/usr/local/airflow/dags/sync_isis_to_kernel.py", line 377, in <listcomp>
    issues_as_kernel = [issue_as_kernel(issue) for issue in issues]
  File "/usr/local/airflow/dags/sync_isis_to_kernel.py", line 206, in issue_as_kernel
    _creation_date = parse_date(issue.publication_date)
  File "/usr/local/airflow/dags/sync_isis_to_kernel.py", line 168, in parse_date
    _date = datetime.strptime(date, "%Y-%m-%d")
TypeError: strptime() argument 1 must be str, not None
[2023-05-05 19:40:16,151] {taskinstance.py:1194} INFO - Marking task as FAILED. dag_id=sync_isis_to_kernel, task_id=process_issues_task, execution_date=20230505T191037, start_date=20230505T194002, end_date=20230505T194016
[2023-05-05 19:40:17,833] {local_task_job.py:102} INFO - Task exited with return code 1

[robertatakenaka]

Foi obtido o arquivo /usr/local/airflow/work/manual__2023-05-05T19:10:37+00:00/json/issue.json com apoio da infra e foi executado o script abaixo para identificar o issue com defeito e corrigir

from xylose.scielodocument import Issue
from datetime import datetime


def parse_date(date):
    """Traduz datas em formato simples ano-mes-dia, ano-mes para
    o formato iso utilizado durantr a persistência do Kernel"""
    _date = None
    try:
        _date = datetime.strptime(date, "%Y-%m-%d")
    except ValueError:
        try:
            _date = datetime.strptime(date, "%Y-%m")
        except ValueError:
            _date = datetime.strptime(date, "%Y")
    return _date


import json
with open('issue.json') as fp:
    d = json.loads(fp.read())
    for i, item in enumerate(d):
        issue = Issue({"issue": item})
        try:
            p = parse_date(issue.publication_date)
        except:
            # identificação do problema
            print(issue.publication_date)
            print(issue.data)
            d[i]["v65"] = [{"_": "20230000"}]

# geração do arquivo corrigido
with open('issue.json', "w", encoding="utf-8") as fp:
    fp.write(json.dumps(d, separators=(',', ':'), ensure_ascii=False))
    

[robertatakenaka]

Item sem o campo v65

data = {'issue': {'v480': [{'_': 'Sociedade Brasileira de Genética'}], 'v200': [{'_': '0'}], 'v541': [{'_': 'BY'}], 'v421': [{'_': 'Genet Mol Biol'}], 'v91': [{'_': '20230505'}], 'v30': [{'_': 'Genet. Mol. Biol.'}], 'v935': [{'_': '1678-4685'}], 'v32': [{'_': '2'}], 'v31': [{'_': '46'}], 'v930': [{'_': 'GMB'}], 'v435': [{'t': 'PRINT', '_': '1415-4757'}, {'t': 'ONLIN', '_': '1678-4685'}], 'v117': [{'_': 'other'}], 'v36': [{'_': '20234'}], 'v35': [{'_': '1415-4757'}], 'v151': [{'_': 'Genet. mol. biol'}], 'v991': [{'_': '1'}], 'v130': [{'_': 'Genetics and Molecular Biology'}], 'v706': [{'_': 'i'}], 'v85': [{'_': 'nd'}], 'mfn': 28944, 'v701': [{'_': '1'}], 'v43': [{'a': '2023', 'c': 'Ribeirão Preto', 'l': 'es', 'n': 'no.2', 't': 'Genet. Mol. Biol.', 'v': 'vol.46'}, {'a': '2023', 'c': 'Ribeirão Preto', 'l': 'pt', 'n': 'no.2', 't': 'Genet. Mol. Biol.', 'v': 'vol.46'}, {'a': '2023', 'c': 'Ribeirão Preto', 'l': 'en', 'n': 'no.2', 't': 'Genet. Mol. Biol.', 'v': 'vol.46'}], 'v700': [{'_': '0'}], 'v49': [{'c': 'GMB020', 'l': 'en', 't': 'Genetics of Microorganisms'}, {'c': 'GMB030', 'l': 'en', 't': 'Animal Genetics'}, {'c': 'GMB040', 'l': 'en', 't': 'Plant Genetics'}, {'c': 'GMB050', 'l': 'en', 't': 'Human and Medical Genetics'}, {'c': 'GMB060', 'l': 'en', 't': 'Mutagenesis'}, {'c': 'GMB140', 'l': 'en', 't': 'Evolution Genetics'}, {'c': 'GMB200', 'l': 'en', 't': 'Evolutionary Genetics'}, {'c': 'GMB240', 'l': 'en', 't': 'Genomics and bioinformatics'}, {'c': 'GMB360', 'l': 'en', 't': 'Erratum'}, {'c': 'GMB380', 'l': 'en', 't': 'Cellular, Molecular and Developmental Genetics'}, {'c': 'GMB460', 'l': 'en', 't': 'Cell, Molecular and Developmental Genetics'}], 'v48': [{'h': 'Sumario', 'l': 'es'}, {'h': 'Sumário', 'l': 'pt'}, {'h': 'Table of Contents', 'l': 'en'}]}

[robertatakenaka]

A ausência é do campo Date ISO da GMB 46 2 (Image do guia sobre preenchimento de issue)