script Python lento que extrae información de una base de datos iSeries

Question

He convertido y actualizado algunos scripts python que exportaban datos de una base de datos iSeries (HTE) a Access para exportar los mismos datos a un fgdb. Mi mayor problema es el tiempo que está tardando en exportar los datos y actualizar la tabla en mi gdb. No estoy seguro de qué es exactamente lo que falta para hacer este Py ir más rápido, pero estoy seguro de que alguien por ahí sabe la respuesta. Estoy usando 10.2. ¡El código se adjunta para su disfrute!

#imports
import pyodbc
import arcpy

#---------------------------------CONNECTION FOR iSERIES 

DATABASE------------------------------------------------------

#Creating a connection string for iSeries
conn_iseries = xxxxxxxxx

#Creating first SQL cursor based on the iSeries connection string
iseries_cursor = conn_iseries.cursor()

#---------------------------------QUERY iSERIES AND INSERT INTO MS ACCESS 

DATABASE-------------------------------------

#Create iSeries SQL query string
sql_qry_iseries = """
                    SELECT  DISTINCT
                            TRIM(WFAHREP.AHALCD) AS FAC_ID, /* Facility ID */
                            TRIM(WFAHREP.AHAOCD) AS FAC_PARENT, /* Parent Facility */
                            TRIM(WFAHREP.AHAGTX) AS FAC_DESC, /* Facility Description */
                            TRIM(WFAHREP.AHAMCD) AS DEPT_RESP, /* Department Responsible */
                            TRIM(WFAHREP.AHAHST) AS FAC_TYPE, /* Facility Type */
                            TRIM(WFAHREP.AHC3CD) AS FAC_SUBTYPE, /* Facility sub Type */
                            TRIM(WFBHREP.BHCVTX) AS SUBTYPE_DESC, /* Facility Subtype Description */
                            TRIM(WFAHREP.AHESCD) AS LOC_CODE, /* General Location Code */
                            TRIM(WFBHREP.BHCNTX) AS TITLE1, /* Facility Code Field 1 Description */
                            TRIM(WFAHREP.AHC8CD) AS CODE1, /* User Code Field 1 */
                            TRIM(WFBIREP1.BICWTX) AS CODEDESC1, /* User Field Code Description */
                            TRIM(WFBHREP.BHCOTX) AS TITLE2, /* Facility Code Field 2 Description */
                            TRIM(WFAHREP.AHC5CD) AS CODE2, /* User Code Field 2 */
                            TRIM(WFBIREP2.BICWTX) AS CODEDESC2, /* User Field Code Description */
                            TRIM(WFBHREP.BHCPTX) AS TITLE3, /* Facility Code Field 3 Description */
                            TRIM(WFAHREP.AHC6CD) AS CODE3, /* User Code Field 3 */
                            TRIM(WFBIREP3.BICWTX) AS CODEDESC3, /* User Field Code Description */
                            TRIM(WFBHREP.BHCQTX) AS TITLE4, /* Facility Code Field 4 Description */
                            TRIM(WFAHREP.AHC7CD) AS CODE4, /* User Code Field 4 */
                            TRIM(WFBIREP4.BICWTX) AS CODEDESC4, /* User Field Code Description */
                            TRIM(WFBHREP.BHCRTX) AS TITLE5, /* Facility Date Field 1 Description */
                            DATEFMT1.DATE_FMT AS DATE5, /* Real Date Field for Facility Date Field 1 

Description */
                            TRIM(WFBHREP.BHCSTX) AS TITLE6, /* Facility Date Field 2 Description */
                            DATEFMT2.DATE_FMT AS DATE6, /* Real Date Field for Facility Date Field 2 

Description */
                            TRIM(WFBHREP.BHCTTX) AS TITLE7, /* Facility Number Field 1 Description */
                            WFAHREP.AHCBNB AS NUMBER7, /* User Numeric Field 1 */
                            TRIM(WFBHREP.BHCUTX) AS TITLE8, /* Facility Numeric Field 2 Description */
                            WFAHREP.AHCCNB AS NUMBER8, /* User Numeric Field 2 */
                            WFAHREP.AHDVST AS ACTIVEFLAG /* Active Flag */
                    FROM
                            HTEDTA.WFAHREP WFAHREP /* Facility Master */

                    LEFT OUTER JOIN
                            HTEDTA.WFBHREP WFBHREP /* Facility Sub-type */
                    ON      WFAHREP.AHAHST=WFBHREP.BHAHST /* Facility type */
                    AND     WFAHREP.AHC3CD=WFBHREP.BHC3CD /* Facility Sub-Type Code */

                    LEFT OUTER JOIN
                            HTEDTA.WFBIREP WFBIREP1 /* Facility Sub-type Codes */
                    ON      WFAHREP.AHAHST = WFBIREP1.BIAHST /* Facility type */
                    AND     WFAHREP.AHC3CD = WFBIREP1.BIC3CD /* Facility Sub-Type Code */
                    AND     WFAHREP.AHB9ST = WFBIREP1.BIB5ST /* User Field 1 ID */
                    AND     WFAHREP.AHC8CD = WFBIREP1.BIC4CD /* User Code Field 1 */

                    LEFT OUTER JOIN
                            HTEDTA.WFBIREP WFBIREP2 /* Facility Sub-type Codes */
                    ON      WFAHREP.AHAHST = WFBIREP2.BIAHST /* Facility type */
                    AND     WFAHREP.AHC3CD = WFBIREP2.BIC3CD /* Facility Sub-Type Code */
                    AND     WFAHREP.AHB6ST = WFBIREP2.BIB5ST /* User Field 2 ID */
                    AND     WFAHREP.AHC5CD = WFBIREP2.BIC4CD /* User Code Field 2 */

                    LEFT OUTER JOIN
                            HTEDTA.WFBIREP WFBIREP3 /* Facility Sub-type Codes */
                    ON      WFAHREP.AHAHST = WFBIREP3.BIAHST /* Facility type */
                    AND     WFAHREP.AHC3CD = WFBIREP3.BIC3CD /* Facility Sub-Type Code */
                    AND     WFAHREP.AHB7ST = WFBIREP3.BIB5ST /* User Field 3 ID */
                    AND     WFAHREP.AHC6CD = WFBIREP3.BIC4CD /* User Code Field 3 */

                    LEFT OUTER JOIN
                            HTEDTA.WFBIREP WFBIREP4 /* Facility Sub-type Codes */
                    ON      WFAHREP.AHAHST = WFBIREP4.BIAHST /* Facility type */
                    AND     WFAHREP.AHC3CD = WFBIREP4.BIC3CD /* Facility Sub-Type Code */
                    AND     WFAHREP.AHB8ST = WFBIREP4.BIB5ST /* User Field 4 ID */
                    AND     WFAHREP.AHC7CD = WFBIREP4.BIC4CD /* User Code Field 4 */

                    LEFT OUTER JOIN
                            COLLIB.DATEFMT DATEFMT1 /* Date Format */
                    ON      WFAHREP.AHAVDT = DATEFMT1.CYYMMDD /* User Date Field 1 */

                    LEFT    OUTER JOIN
                            COLLIB.DATEFMT DATEFMT2 /* Date Format */
                    ON      WFAHREP.AHAWDT = DATEFMT2.CYYMMDD /* User Date Field 2 */

                    WHERE
                            WFAHREP.AHALCD NOT LIKE '%''%' /* Facility ID NOT HAVING ANY SINGLE QUOTE */

                """

#Execute the sql string
iseries_cursor.execute(sql_qry_iseries)

#Main while loop - get the SQL data
while 1:

    #Fetch one row from the cursor1 to the row1
    iseries_row = iseries_cursor.fetchone()

    #if row1 does not have any record/data then break the 'while' loop
    if not iseries_row:
        break

    #-------------------------------------------------------------------
    #Assigning the row field data to veriables

    #Facility ID
    FAC_ID1 = str(iseries_row.FAC_ID)
    #Parent Facility
    FAC_PARENT1 = str(iseries_row.FAC_PARENT)

    #Facility Description
    #If the Facility Description has a single quote (') inside, then Access database SQL will not accept that
    FAC_DESC1 = str(iseries_row.FAC_DESC)
    #Replacing all tthe single quote (if any) with two single quote
    NEW_FAC_DESC1 = FAC_DESC1.replace("'","''")

    #Department Responsible
    DEPT_RESP1 = str(iseries_row.DEPT_RESP)
    #Facility Type
    FAC_TYPE1 = str(iseries_row.FAC_TYPE)
    #Facility sub Type
    FAC_SUBTYPE1 = str(iseries_row.FAC_SUBTYPE)

    #Facility Subtype Description
    #If the Facility Subtype Description has a single quote (') inside, then Access database SQL will not 

accept that
    SUBTYPE_DESC1 = str(iseries_row.SUBTYPE_DESC)
    #Replacing all tthe single quote (if any) with two single quote
    NEW_SUBTYPE_DESC1 = SUBTYPE_DESC1.replace("'","''")

    #General Location Code
    LOC_CODE1 = str(iseries_row.LOC_CODE)

    #Facility Code Field 1 Description
    TITLE1 = str(iseries_row.TITLE1)

    #User Code Field 1
    CODE1 = str(iseries_row.CODE1)

    #User Field Code Description
    #If the Facility Subtype Description has a single quote (') inside, then Access database SQL will not 

accept that
    CODEDESC1 = str(iseries_row.CODEDESC1)
    #Replacing all tthe single quote (if any) with two single quote
    NEWCODEDESC1 = CODEDESC1.replace("'","''")

    #Facility Code Field 2 Description
    TITLE2 = str(iseries_row.TITLE2)

    #User Code Field 2
    CODE2 = str(iseries_row.CODE2)

    #User Field Code Description
    #If the Facility Subtype Description has a single quote (') inside, then Access database SQL will not 

accept that
    CODEDESC2 = str(iseries_row.CODEDESC2)
    #Replacing all tthe single quote (if any) with two single quote
    NEWCODEDESC2 = CODEDESC2.replace("'","''")

    #Facility Code Field 3 Description
    TITLE3 = str(iseries_row.TITLE3)

    #User Code Field 3
    CODE3 = str(iseries_row.CODE3)

    #User Field Code Description
    #If the Facility Subtype Description has a single quote (') inside, then Access database SQL will not 

accept that
    CODEDESC3 = str(iseries_row.CODEDESC3)
    #Replacing all tthe single quote (if any) with two single quote
    NEWCODEDESC3 = CODEDESC3.replace("'","''")

    #Facility Code Field 4 Description
    TITLE4 = str(iseries_row.TITLE4)

    #User Code Field 4
    CODE4 = str(iseries_row.CODE4)

    #User Field Code Description
    #If the Facility Subtype Description has a single quote (') inside, then Access database SQL will not 

accept that
    CODEDESC4 = str(iseries_row.CODEDESC4)
    #Replacing all tthe single quote (if any) with two single quote
    NEWCODEDESC4 = CODEDESC4.replace("'","''")

    #Facility Date Field 1 Description
    TITLE5 = str(iseries_row.TITLE5)

    #Real Date Field for Facility Date Field 1 Description
    #If the date has a None value returned then change it to 'Null' as Access SQL date field understands Null 

Date
    if str(iseries_row.DATE5) == 'None':
    #   MSAccess Data import --- DATE5 = 'Null'
        DATE5 = 0
    #Else assign the date value to the variable
    else:
    #   MSAccess Date Formating --- DATE5 = '#' + str(iseries_row.DATE5) + '#'
        DATE5 = str(iseries_row.DATE5)

    #Facility Date Field 2 Description
    TITLE6 = str(iseries_row.TITLE6)

    #Real Date Field for Facility Date Field 2 Description
    #If the date has a None value returned then change it to 'Null' as Access SQL date field understands Null 

Date
    if str(iseries_row.DATE6) == 'None':
    #   MSAccess Data import --- DATE6 = 'Null'
        DATE6 = 0
    #Else assign the date value to the variable
    else:
    #   MSAccess Date Formating --- DATE6 = '#' + str(iseries_row.DATE6) + '#'
        DATE6 = str(iseries_row.DATE6)

    #Facility Number Field 1 Description
    TITLE7 = str(iseries_row.TITLE7)

    #User Numeric Field 1
    #If the numeric field returns 'None' value, we will change it to '0' (Zero)
    if str(iseries_row.NUMBER7) == 'None':
        NUMBER7 = '0'
    #Else keep the returned value
    else:
        NUMBER7 = str(iseries_row.NUMBER7)

    #Facility Number Field 2 Description
    TITLE8 = str(iseries_row.TITLE8)

    #User Numeric Field 2
    #If the numeric field returns 'None' value, we will change it to '0' (Zero)
    if str(iseries_row.NUMBER8) == 'None':
        NUMBER8 = '0'
    #Else keep the returned value
    else:
        NUMBER8 = str(iseries_row.NUMBER8)

    #Active Flag
    ACTIVEFLAG = str(iseries_row.ACTIVEFLAG)

    #ArcPy insert code goes here!
    # Create insert cursor for table
    #
    cursor = arcpy.da.InsertCursor("D:\GIS\Users\xxxxx\xxxxxxx\U_test.gdb\WFAHREP",
                               ["FAC_ID", "FAC_PARENT", "FAC_DESC", "DEPT_RESP",
                               "FAC_TYPE", "FAC_SUBTYP", "SUBTYPDESC", "LOC_CODE",
                               "TITLE1", "CODE1", "CODEDESC1", "TITLE2", "CODE2",
                               "CODEDESC2", "TITLE3", "CODE3", "CODEDESC3", "TITLE4",
                               "CODE4", "CODEDESC4", "TITLE5", "DATE5", "TITLE6",
                               "DATE6", "TITLE7", "NUMBER7", "TITLE8", "NUMBER8",
                               "ACTIVEFLAG"])

    #Inserting into gdb
    cursor.insertRow([FAC_ID1, FAC_PARENT1, NEW_FAC_DESC1, DEPT_RESP1, FAC_TYPE1, FAC_SUBTYPE1, 

NEW_SUBTYPE_DESC1,
    LOC_CODE1, TITLE1, CODE1, NEWCODEDESC1, TITLE2, CODE2, NEWCODEDESC2, TITLE3, CODE3,NEWCODEDESC3, TITLE4,
    CODE4, NEWCODEDESC4, TITLE5, DATE5, TITLE6, DATE6, TITLE7, NUMBER7, TITLE8, NUMBER8, ACTIVEFLAG])

#Close the iSeries SQL cursor
iseries_cursor.close()

# Delete arcpy cursor object
del cursor

print 'WFAHREP table is updated.'

Answer 1

Para determinar en qué parte de su script largo se encuentran sus cuellos de botella de rendimiento, le recomiendo que utilice la herramienta de Python time de forma similar a como lo hice en este módulo responder a otra pregunta.

Una vez que sepa dónde están los cuellos de botella, podrá aislarlos en fragmentos de código de prueba que podrá utilizar para intentar desarrollar patrones de codificación alternativos (más rápidos).

Cada uno de esos fragmentos de código podría convertirse en el centro de un nuevo Q & A aquí, pero sospecho que en el proceso de extracción que bien puede encontrar la manera de acelerar o ser capaz de concluir si realmente está golpeando un ArcPy / Python / iSeries limitación de rendimiento.