Unable to use an existing Hive permanent UDF from Spark SQL

Unable to use an existing Hive permanent UDF from Spark SQL

Add Comment


  • 2 Answer(s)

    Here the problem is version Spark 2.0 is not able to execute the functions whose JARs are located on HDFS.

    Spark SQL: Thriftserver unable to run a registered Hive UDTF

    With jar path pointing to a local edge-node path,define the function as a temporary function in Spark job . Then call the function in same Spark job.

    CREATE TEMPORARY FUNCTION functionName as 'com.test.HiveUDF' USING JAR '/user/home/dir1/functions.jar'
    
    
    Answered on January 4, 2019.
    Add Comment

    In select function cannot be  directly called.

    Like oracle create some dumy table.

    CREATE TABLE dual (dummy STRING);
    
    

    In path ‘/path/to/textfile/dual.txt’¬†load data local to overwrite into table dual;

    SELECT normaliseURL('value') from dual;
    

    or else

    SELECT * from normaliseURL('value')
    
    Answered on January 4, 2019.
    Add Comment


  • Your Answer

    By posting your answer, you agree to the privacy policy and terms of service.