Merge branch 'dev' into extended

Author: asandikci

.github/CONTRIBUTING.md

@@ -0,0 +1,84 @@
+PipeBender contribution guidelines
+===============================
+
+## Crash reporting
+
+Report crashes through the **automated crash report system** of PipeBender.
+This way all the data needed for debugging is included in your bug report for GitHub.
+You'll see *exactly* what is sent, be able to add **your comments**, and then send it.
+
+## Issue reporting/feature requests
+
+* **Already reported**? Browse the existing issues in [NewPipe](https://github.com/TeamNewPipe/NewPipe/issues), [Tubular] and [PipeBender] repositories to make sure your issue/feature hasn't been reported/requested.
+* **Already fixed**? Check whether your issue/feature is already fixed/implemented.
+* **Still relevant**? Check if the issue still exists in the latest release/beta version.
+* **Can you fix it**? If you are an Android/Java developer, you are always welcome to fix an issue or implement a feature yourself. PRs welcome! See [Code contribution](#code-contribution) for more info.
+* **Is it in English**? Issues in other languages will be ignored unless someone translates them.
+* **Is it one issue**? Multiple issues require multiple reports, that can be linked to track their statuses.
+* **The template**: Fill it out, everyone wins. Your issue has a chance of getting fixed.
+
+
+## Translation
+
+* NewPipe is translated via [Weblate](https://hosted.weblate.org/projects/newpipe/strings/). Log in there with your GitHub account, or register.
+  * as MaintainTeam organization we will set up a distinct weblate page later
+* Add the language you want to translate if it is not there already: see [How to add a new language](https://github.com/TeamNewPipe/NewPipe/wiki/How-to-add-a-new-language-to-NewPipe) in the wiki.
+* NewPipe uses the [PrettyTime](https://github.com/ocpsoft/prettytime) library to display localized versions of dates and times. It needs to be translated, too. Read [these instructions to add a new language](https://www.ocpsoft.org/prettytime/#section-14) and [this issue](https://github.com/TeamNewPipe/NewPipe/issues/9134) for more info.
+
+## Code contribution
+
+### Guidelines
+
+* Stick to NewPipe's *style conventions* of [checkStyle](https://github.com/checkstyle/checkstyle) and [ktlint](https://github.com/pinterest/ktlint). They run each time you build the project.
+* Stick to [F-Droid contribution guidelines](https://f-droid.org/wiki/page/Inclusion_Policy).
+* In particular **do not bring non-free software** (e.g. binary blobs) into the project. Make sure you do not introduce any closed-source library from Google.
+
+### Before starting development
+
+* If you want to help out with an existing bug report or feature request, **leave a comment** on that issue saying you want to try your hand at it.
+* If there is no existing issue for what you want to work on, **open a new one**  describing the changes you are planning to introduce. This gives the team and the community a chance to give **feedback** before you spend time on something that is already in development, should be done differently, or should be avoided completely.
+* Please show **intention to maintain your features** and code after you contribute a PR. Unmaintained code is a hassle for core developers. If you do not intend to maintain features you plan to contribute, please rethink your submission, or clearly state that in the PR description.
+* Create PRs that cover only **one specific issue/solution/bug**. Do not create PRs that are huge monoliths and could have been split into multiple independent contributions.
+* NewPipe uses [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor) to fetch data from services. If you need to change something there, you must test your changes in NewPipe. Telling NewPipe to use your extractor version can be accomplished by editing the `app/build.gradle` file: the comments under the "NewPipe libraries" section of `dependencies` will help you out.
+  * We use [LastPipeExtractor](https://github.com/MaintainTeam/LastPipeExtractor)
+
+### Creating a Pull Request (PR)
+
+* Make changes on a **separate branch** with a meaningful name, not on the _master_ branch or the _dev_ branch. This is commonly known as *feature branch workflow*. You may then send your changes as a pull request (PR) on GitHub.
+* Please **test** (compile and run) your code before submitting changes! Ideally, provide test feedback in the PR description. Untested code will **not** be merged!
+* Respond if someone requests changes or otherwise raises issues about your PRs.
+* Try to figure out yourself why builds on our CI fail.
+* Make sure your PR is **up-to-date** with the rest of the code. Often, a simple click on "Update branch" will do the job, but if not, you must *rebase* your branch on the `dev` branch manually and resolve the conflicts on your own. You can find help [on the wiki](https://github.com/TeamNewPipe/NewPipe/wiki/How-to-merge-a-PR). Doing this makes the maintainers' job way easier.
+
+## IDE setup & building the app
+
+### Basic setup
+
+NewPipe is developed using [Android Studio](https://developer.android.com/studio/). Learn more about how to install it and how it works in the [official documentation](https://developer.android.com/studio/intro). In particular, make sure you have accepted Android Studio's SDK licences. Once Android Studio is ready, setting up the NewPipe project is fairly simple:
+- Clone the NewPipe repository with `git clone https://github.com/TeamNewPipe/NewPipe.git` (or use the link from your own fork, if you want to open a PR).
+- Open the folder you just cloned with Android Studio.
+- Build and run it just like you would do with any other app, with the green triangle in the top bar.
+
+You may find [SonarLint](https://www.sonarlint.org/intellij)'s **inspections** useful in helping you to write good code and prevent bugs.
+
+### checkStyle setup
+
+The [checkStyle](https://github.com/checkstyle/checkstyle) plugin verifies that Java code abides by the project style. It runs automatically each time you build the project. If you want to view errors directly in the editor, instead of having to skim through the build output, you can install an Android Studio plugin:
+- Go to `File -> Settings -> Plugins`, search for `checkstyle` and install `CheckStyle-IDEA`.
+- Go to `File -> Settings -> Tools -> Checkstyle`.
+- Add NewPipe's configuration file by clicking the `+` in the right toolbar of the "Configuration File" list.
+- Under the "Use a local Checkstyle file" bullet, click on `Browse` and, enter `checkstyle` folder under the project's root path and pick the file named `checkstyle.xml`.
+- Enable "Store relative to project location" so that moving the directory around does not create issues.
+- Insert a description in the top bar, then click `Next` and then `Finish`.
+- Activate the configuration file you just added by enabling the checkbox on the left.
+- Click `Ok` and you are done.
+
+### ktlint setup
+
+The [ktlint](https://github.com/pinterest/ktlint) plugin does the same job as checkStyle for Kotlin files. Installing the related plugin is as simple as going to `File -> Settings -> Plugins`, searching for `ktlint` and installing `Ktlint (unofficial)`.
+
+## Communication
+
+* You can use a Matrix account to join the NewPipe channel at [#newpipe:matrix.newpipe-ev.de](https://matrix.to/#/#newpipe:matrix.newpipe-ev.de). Some convenient clients, available both for phone and desktop, are listed at that link.
+* Alternatively, the #newpipe channel on Libera Chat (`ircs://irc.libera.chat:6697/newpipe`) can also be joined, as it is bridged to the Matrix room. [Click here for webchat](https://web.libera.chat/#newpipe)!
+* You can post your suggestions, changes, ideas etc. on either GitHub or Matrix (including via IRC).

.github/workflows/build-release-apk.yml

@@ -14,7 +14,7 @@ jobs:
       - uses: actions/setup-java@v4
         with:
           distribution: 'temurin'
-          java-version: '17'
+          java-version: '21'
           cache: 'gradle'
 
       - name: "Build release APK"

.github/workflows/release.yml

@@ -176,12 +176,12 @@ jobs:
           cd "${{ github.workspace }}/output/"
           version=$(cat version.num)
           echo $version
-          gh release create "v${version}" --title "${{ inputs.title }}" --notes-file "./changelog.md" --prerelease="true" --repo MaintainTeam/LastPipeBender
+          gh release create "v${version}" --title "${{ inputs.title }}" --notes-file "./changelog.md" --prerelease="true" --discussion-category "Announcements" --repo MaintainTeam/LastPipeBender
           gh release upload "v${version}" $(echo $(find . -name '*.apk' -type f -exec basename \{} \;) checksums.txt) --repo MaintainTeam/LastPipeBender
 
       - name: Archive reports for job
         uses: actions/upload-artifact@v4
         with:
           name: reports
           path: '*/build/reports'
-        if: ${{ always() }}
+        if: ${{ always() }}

README.md

@@ -47,6 +47,9 @@ This is the SHA fingerprint of LastPipeBender's signing key to verify downloaded
 1B:00:8D:64:BB:95:AB:47:74:D6:8B:87:F2:2B:8B:E9:A2:72:F4:92:4D:F5:20:29:D7:E6:18:38:35:D9:18:CC
 ```
 
+You should also check checksums in releases. Also you can compare checksums with GitHub Actions notices to ensure the apk file built with CI
+
+
 ### Project Management
 
 ```mermaid

app/build.gradle

@@ -23,9 +23,9 @@ android {
         if (System.properties.containsKey('versionCodeOverride')) {
             versionCode System.getProperty('versionCodeOverride') as Integer
         } else {
-            versionCode 1002
+            versionCode 1003
         }
-        versionName "0.27.5"
+        versionName "0.27.6"
         if (System.properties.containsKey('versionNameSuffix')) {
             versionNameSuffix System.getProperty('versionNameSuffix')
         }
@@ -220,7 +220,7 @@ dependencies {
     implementation 'com.github.TeamNewPipe:nanojson:1d9e1aea9049fc9f85e68b43ba39fe7be1c1f751'
     // WORKAROUND: if you get errors with the NewPipeExtractor dependency, replace `v0.24.3` with
     // the corresponding commit hash, since JitPack is sometimes buggy
-    implementation 'com.github.MaintainTeam:LastPipeExtractor:d6fd0c38ca6ed4602cc8b3b0bdd012eb24d8e60c'
+    implementation 'com.github.MaintainTeam:LastPipeExtractor:6bf7594fbc59ba2c6372b71addbe31aa2e33118f'
     implementation 'com.github.TeamNewPipe:NoNonsense-FilePicker:5.0.0'
 
 /** Checkstyle **/
@@ -253,6 +253,7 @@ dependencies {
     implementation "androidx.work:work-runtime-ktx:${androidxWorkVersion}"
     implementation "androidx.work:work-rxjava3:${androidxWorkVersion}"
     implementation 'com.google.android.material:material:1.11.0'
+    implementation "androidx.webkit:webkit:1.9.0"
 
 /** Third-party libraries **/
     // Instance state boilerplate elimination

app/src/main/assets/po_token.html

@@ -0,0 +1,127 @@
+<!DOCTYPE html>
+<html lang="en"><head><title></title><script>
+/**
+ * Factory method to create and load a BotGuardClient instance.
+ * @param options - Configuration options for the BotGuardClient.
+ * @returns A promise that resolves to a loaded BotGuardClient instance.
+ */
+function loadBotGuard(challengeData) {
+  this.vm = this[challengeData.globalName];
+  this.program = challengeData.program;
+  this.vmFunctions = {};
+  this.syncSnapshotFunction = null;
+
+  if (!this.vm)
+    throw new Error('[BotGuardClient]: VM not found in the global object');
+
+  if (!this.vm.a)
+    throw new Error('[BotGuardClient]: Could not load program');
+
+  const vmFunctionsCallback = function (
+    asyncSnapshotFunction,
+    shutdownFunction,
+    passEventFunction,
+    checkCameraFunction
+  ) {
+    this.vmFunctions = {
+      asyncSnapshotFunction: asyncSnapshotFunction,
+      shutdownFunction: shutdownFunction,
+      passEventFunction: passEventFunction,
+      checkCameraFunction: checkCameraFunction
+    };
+  };
+
+  this.syncSnapshotFunction = this.vm.a(this.program, vmFunctionsCallback, true, this.userInteractionElement, function () {/** no-op */ }, [ [], [] ])[0]
+
+  // an asynchronous function runs in the background and it will eventually call
+  // `vmFunctionsCallback`, however we need to manually tell JavaScript to pass
+  // control to the things running in the background by interrupting this async
+  // function in any way, e.g. with a delay of 1ms. The loop is most probably not
+  // needed but is there just because.
+  return new Promise(function (resolve, reject) {
+    i = 0
+    refreshIntervalId = setInterval(function () {
+      if (!!this.vmFunctions.asyncSnapshotFunction) {
+        resolve(this)
+        clearInterval(refreshIntervalId);
+      }
+      if (i >= 10000) {
+        reject("asyncSnapshotFunction is null even after 10 seconds")
+        clearInterval(refreshIntervalId);
+      }
+      i += 1;
+    }, 1);
+  })
+}
+
+/**
+ * Takes a snapshot asynchronously.
+ * @returns The snapshot result.
+ * @example
+ * ```ts
+ * const result = await botguard.snapshot({
+ *   contentBinding: {
+ *     c: "a=6&a2=10&b=SZWDwKVIuixOp7Y4euGTgwckbJA&c=1729143849&d=1&t=7200&c1a=1&c6a=1&c6b=1&hh=HrMb5mRWTyxGJphDr0nW2Oxonh0_wl2BDqWuLHyeKLo",
+ *     e: "ENGAGEMENT_TYPE_VIDEO_LIKE",
+ *     encryptedVideoId: "P-vC09ZJcnM"
+ *    }
+ * });
+ *
+ * console.log(result);
+ * ```
+ */
+function snapshot(args) {
+  return new Promise(function (resolve, reject) {
+    if (!this.vmFunctions.asyncSnapshotFunction)
+      return reject(new Error('[BotGuardClient]: Async snapshot function not found'));
+
+    this.vmFunctions.asyncSnapshotFunction(function (response) { resolve(response) }, [
+      args.contentBinding,
+      args.signedTimestamp,
+      args.webPoSignalOutput,
+      args.skipPrivacyBuffer
+    ]);
+  });
+}
+
+function runBotGuard(challengeData) {
+  const interpreterJavascript = challengeData.interpreterJavascript.privateDoNotAccessOrElseSafeScriptWrappedValue;
+
+  if (interpreterJavascript) {
+    new Function(interpreterJavascript)();
+  } else throw new Error('Could not load VM');
+
+  const webPoSignalOutput = [];
+  return loadBotGuard({
+    globalName: challengeData.globalName,
+    globalObj: this,
+    program: challengeData.program
+  }).then(function (botguard) {
+    return botguard.snapshot({ webPoSignalOutput: webPoSignalOutput })
+  }).then(function (botguardResponse) {
+    return { webPoSignalOutput: webPoSignalOutput, botguardResponse: botguardResponse }
+  })
+}
+
+function obtainPoToken(webPoSignalOutput, integrityToken, identifier) {
+  const getMinter = webPoSignalOutput[0];
+
+  if (!getMinter)
+    throw new Error('PMD:Undefined');
+
+  const mintCallback = getMinter(integrityToken);
+
+  if (!(mintCallback instanceof Function))
+    throw new Error('APF:Failed');
+
+  const result = mintCallback(identifier);
+
+  if (!result)
+    throw new Error('YNJ:Undefined');
+
+  if (!(result instanceof Uint8Array))
+    throw new Error('ODM:Invalid');
+
+  return result;
+}
+</script></head><body></body></html>

app/src/main/java/org/schabi/newpipe/App.java

@@ -17,6 +17,7 @@
 import org.schabi.newpipe.error.ReCaptchaActivity;
 import org.schabi.newpipe.extractor.NewPipe;
 import org.schabi.newpipe.extractor.downloader.Downloader;
+import org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeStreamExtractor;
 import org.schabi.newpipe.ktx.ExceptionUtils;
 import org.schabi.newpipe.settings.NewPipeSettings;
 import org.schabi.newpipe.util.BridgeStateSaverInitializer;
@@ -26,6 +27,7 @@
 import org.schabi.newpipe.util.image.ImageStrategy;
 import org.schabi.newpipe.util.image.PicassoHelper;
 import org.schabi.newpipe.util.image.PreferredImageQuality;
+import org.schabi.newpipe.util.potoken.PoTokenProviderImpl;
 
 import java.io.IOException;
 import java.io.InterruptedIOException;
@@ -118,6 +120,8 @@ public void onCreate() {
                 && prefs.getBoolean(getString(R.string.show_image_indicators_key), false));
 
         configureRxJavaErrorHandler();
+
+        YoutubeStreamExtractor.setPoTokenProvider(PoTokenProviderImpl.INSTANCE);
     }
 
     @Override

app/src/main/java/org/schabi/newpipe/DownloaderImpl.java

@@ -137,46 +137,42 @@ public Response execute(@NonNull final Request request)
         }
 
         final okhttp3.Request.Builder requestBuilder = new okhttp3.Request.Builder()
-                .method(httpMethod, requestBody).url(url)
+                .method(httpMethod, requestBody)
+                .url(url)
                 .addHeader("User-Agent", USER_AGENT);
 
         final String cookies = getCookies(url);
         if (!cookies.isEmpty()) {
             requestBuilder.addHeader("Cookie", cookies);
         }
 
-        for (final Map.Entry<String, List<String>> pair : headers.entrySet()) {
-            final String headerName = pair.getKey();
-            final List<String> headerValueList = pair.getValue();
+        headers.forEach((headerName, headerValueList) -> {
+            requestBuilder.removeHeader(headerName);
+            headerValueList.forEach(headerValue ->
+                    requestBuilder.addHeader(headerName, headerValue));
+        });
+
+        try (
+                okhttp3.Response response = client.newCall(requestBuilder.build()).execute()
+        ) {
+            if (response.code() == 429) {
+                throw new ReCaptchaException("reCaptcha Challenge requested", url);
+            }
 
-            if (headerValueList.size() > 1) {
-                requestBuilder.removeHeader(headerName);
-                for (final String headerValue : headerValueList) {
-                    requestBuilder.addHeader(headerName, headerValue);
+            String responseBodyToReturn = null;
+            try (ResponseBody body = response.body()) {
+                if (body != null) {
+                    responseBodyToReturn = body.string();
                 }
-            } else if (headerValueList.size() == 1) {
-                requestBuilder.header(headerName, headerValueList.get(0));
             }
 
+            final String latestUrl = response.request().url().toString();
+            return new Response(
+                    response.code(),
+                    response.message(),
+                    response.headers().toMultimap(),
+                    responseBodyToReturn,
+                    latestUrl);
         }
-
-        final okhttp3.Response response = client.newCall(requestBuilder.build()).execute();
-
-        if (response.code() == 429) {
-            response.close();
-
-            throw new ReCaptchaException("reCaptcha Challenge requested", url);
-        }
-
-        final ResponseBody body = response.body();
-        String responseBodyToReturn = null;
-
-        if (body != null) {
-            responseBodyToReturn = body.string();
-        }
-
-        final String latestUrl = response.request().url().toString();
-        return new Response(response.code(), response.message(), response.headers().toMultimap(),
-                responseBodyToReturn, latestUrl);
     }
 }