sql >> Database teknologi >  >> RDS >> PostgreSQL

Sqoop på Hadoop:NoSuchMethodError:com.google.common.base.Stopwatch.createStarted()

Efter noget yderligere research kan jeg se, at Hadoop 2.x tilsidesætter klassestien, så løsningen er at oprette en uberjar og send det videre til hadoop.

Jeg ændrede også til at bruge den specifikke sqoop jar til hadoop260 i stedet.

Så jeg oprettede en pom.xml fil, kørte maven package på den for at generere uberjar:

<?xml version="1.0" encoding="UTF-8"?>
<!-- SEE: https://cloud.google.com/blog/products/data-analytics/managing-java-dependencies-apache-spark-applications-cloud-dataproc -->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>
    <groupId>org.example.exporter</groupId>
    <artifactId>example-exporter-postgresql</artifactId>
    <version>0.0.1</version>
    <!-- YOUR_DEPENDENCIES -->
    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.sqoop/sqoop -->
        <dependency>
            <groupId>org.apache.sqoop</groupId>
            <artifactId>sqoop</artifactId>
            <version>1.4.7</version>
            <classifier>hadoop260</classifier>
        </dependency>
        <!-- https://mvnrepository.com/artifact/postgresql/postgresql -->
        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <version>42.2.9</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.google.cloud.sql/postgres-socket-factory -->
        <dependency>
            <groupId>com.google.cloud.sql</groupId>
            <artifactId>postgres-socket-factory</artifactId>
            <version>1.0.15</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.avro/avro-tools -->
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-tools</artifactId>
            <version>1.9.1</version>
        </dependency>

    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>org.apache.sqoop.Sqoop</mainClass>
                                </transformer>
                            </transformers>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/maven/**</exclude>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <relocations>
                                <relocation>
                                    <pattern>com</pattern>
                                    <shadedPattern>repackaged.com.google.common</shadedPattern>
                                    <includes>
                                        <include>com.google.common.**</include>
                                    </includes>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>



  1. Hvordan ændrer jeg datoformatet i Postgres?

  2. Flere showplan-forbedringer? Ja tak!

  3. Float eller decimal for priser?

  4. Har MySQL tidsbaserede triggere?